If one is able to improve how people are matched, it would bring about a huge amount of utility for the entire world.

People would be happier, they would be more productive, there would be less of the divorce-related waste. Being in a happy couple also means you are less distracted by conflict in the house, which leads to people better able to develop themselves and achieve their personal goals. You can keep adding to the direct benefits of being in a good pairing versus a bad pairing.

But it doesn't stop there. If we accept that better matched parents raise their children better, then you are looking at a huge improvement in the psychological health of the next generation of humans. And well-raised humans are more likely to match better with each other...

Under this light, it strikes me as vastly suboptimal that people today will get married to the best option available in their immediate environment when they reach the right age.

The cutting-edge online dating sites base their suggestions on a very limited list of questions. But each of us outputs huge amounts of data, many of them available through APIs on the web. Favourite books, movies, sleep patterns, browsing history, work history, health data, and so much more. We should be using that data to form good hypotheses on how to better match people. I'm actually shocked at the underinvestment in this area as a legitimate altruistic cause.

If an altruistic group of numbers-inclined people was to start working together to improve the world in a non-existential risk reducing kind of way, it strikes me that a dating site may be a fantastic thing to try. On the off-chance it actually produces real results, Applied Rationality will also have a great story of how it improved the world. And, you know, it might even make money.

Thoughts? Any better options?

I've noticed I don't read 'Main' posts anymore.

When I come to LW, I click to the Discussion almost instinctively. I'd estimate it has been four weeks since I've looked at Main. I sometimes read new Slate Star Codex posts (super good stuff, if you are unfamiliar) from LW's sidebar. I sometimes notice interesting-sounding 'Recent Comments' and click on them.

My initial thought is that I don't feel compelled to read Main posts because they are the LW-approved ideas, and I'm not super interested in listening to a bunch of people agreeing with another. Maybe that is a caricature, not sure.

Anyone else Discussion-centric in their LW use?

Also, the Meetup stuff is annoying noise. I'm very sympathetic if placing it among posts helps to drive attendance. By all means, continue if it helps your causes. But it feels spammy to me.

How do you pick a career if your goal is to maximize your income (technically, maximize the expected value of some function of your income)? The sort of standard answer is "comparative advantage", but it's unclear to me how to apply that concept in practice. For example how much demand there is for each kind of job is obviously very important, but how do you take that into consideration, exactly? I've been thinking about this and came up with the following. I'd be interested in any improvements or alternative ideas.

1. For each career under consideration, estimate your potential income ranking or percentile within that career if you went into it (as a probability distribution).
2. For each career, estimate its income distribution (how much will the top earner make, how much will the second highest earner make, etc.).
3. From 1 and 2, obtain a probability distribution of your income within each career.
4. Pick the career with maximum expected utility.

So I have the typical of introvert/nerd problem of being shy about meeting people one-on-one, because I'm afraid of not being able to come up with anything to say and lots of awkwardness resulting. (Might have something to do with why I've typically tended to date talkative people...)

Now I'm pretty sure that there must exist some excellent book or guide or blog post series or whatever that's aimed at teaching people how to actually be a good conversationalist. I just haven't found it. Recommendations?

I noticed recently that one of the mental processes that gets in the way of my proper thinking is an urge to instantly answer a question then spend the rest of my time trying to justify that knee-jerk answer.

For example, I saw a post recently asking whether chess or poker was more popular worldwide. For some reason I wanted to say "obviously x is more popular," but I realized that I don't actually know. And if I avoid that urge to answer the question instantly, it's much easier for me to keep my ego out of issues and to investigate things properly...including making it easier for me recognize things that I don't know and acknowledge that I don't know them.

Is there a formal name for this type of bias or behavior pattern? It would let me search up some Sequence posts or articles to read.

In "The Fall and Rise of Formal Methods", Peter Amey gives a pretty good description of how I expect things to play out w.r.t. Friendly AI research:

Good ideas sometimes come before their time. They may be too novel for their merit to be recognised. They may be seen to threaten some party’s self interest. They may be seen as simply too hard to adopt. These premature good ideas are often swept into corners and, the world, breathing a sigh of relief, gets on with whatever it was up to before they came along. Fortunately not all good ideas wither. Some are kept alive by enthusiasts, who seize every opportunity to show that they really are good ideas. In some cases the world eventually catches up and the original premature good idea, honed by its period of isolation, bursts forth as the new normality (sometimes with its original critics claiming it was all their idea in the first place!).

Formal methods (and I’ll outline in more detail what I mean by ‘formal methods’ shortly) are a classic example of early oppression followed by later resurgence. They arrived on the scene at a time when developers were preoccupied with trying to squeeze complex functionality into hardware with too little memory and too slow processors.

...[But now] formal methods... are on the rise. And why not? What is the alternative? If we don’t use them, then our tool box contains just one spanner labelled ‘test’. We know that for anything that justifies the label ‘critical’, testing will never be enough...

For the curious, Amey also wrote a nice overview of successes and failures in formal methods.

Here is another logic puzzle. I did not write this one, but I really like it.

Imagine you have a circular cake, that is frosted on the top. You cut a d degree slice out of it, and then put it back, but rotated so that it is upside down. Now, d degrees of the cake have frosting on the bottom, while 360 minus d degrees have frosting on the top. Rotate the cake d degrees, take the next slice, and put it upside down. Now, assuming the d is less than 180, 2d degrees of the cake will have frosting on the bottom.

If d is 60 degrees, then after you repeat this procedure, flipping a single slice and rotating 6 times, all the frosting will be on the bottom. If you repeat the procedure 12 times, all of the frosting will be back on the top of the cake.

For what values of d does the cake eventually get back to having all the frosting on the top?

Solution can be found in the comments here.

Introduction I suspected that the type of stuff that gets posted in Rationality Quotes reinforces the mistaken way of throwing about the word rational. To test this, I set out to look at the first twenty rationality quotes in the most recent RQ thread. In the end I only looked at the first ten because it was taking more time and energy than would permit me to continue past that. (I'd only seen one of them before, namely the one that prompted me to make this comment.)

A look at the quotes

In our large, anonymous society, it's easy to forget moral and reputational pressures and concentrate on legal pressure and security systems. This is a mistake; even though our informal social pressures fade into the background, they're still responsible for most of the cooperation in society.

There might be an intended, implicit lesson here that would systematically improve thinking, but without more concrete examples and elaboration (I'm not sure what the exact mistake being pointed to is), we're left guessing what it might be. In cases like this where it's not clear, it's best to point out explicitly what the general habit of thought (cognitive algorithm) is that should be corrected, and how one should correct it, rather than to point in the vague direction of something highly specific going wrong.

As the world becomes more addictive, the two senses in which one can live a normal life will be driven ever further apart. One sense of "normal" is statistically normal: what everyone else does. The other is the sense we mean when we talk about the normal operating range of a piece of machinery: what works best.

These two senses are already quite far apart. Already someone trying to live well would seem eccentrically abstemious in most of the US. That phenomenon is only going to become more pronounced. You can probably take it as a rule of thumb from now on that if people don't think you're weird, you're living badly.

Without context, I'm struggling to understand the meaning of this quote, too. The Paul Graham article it appears in, after a quick skim, does not appear to be teaching a general lesson about how to think; rather it appears to be making a specific observation. I don't feel like I've learned about a bad cognitive habit I had by reading this, or been taught a new useful way to think.

If you're expecting the world to be fair with you because you are fair, you are fooling yourself. That's like expecting a lion not to eat you because you didn't eat him.

Although this again seems like it's vague enough that the range of possible interpretations is fairly broad, I feel like this is interpretable into useful advice. It doesn't make a clear point about habits of thought, though, and I had to consciously try to make up a plausible general lesson for it (just world fallacy), that I probably wouldn't have been able to think up if I didn't already know that general lesson.

He says we could learn a lot from primitive tribes. But they could learn a lot more from us!

I understand and like this quote. It feels like this quote is an antidote to a specific type of thought (patronising signalling of reverence for the wisdom of primitive tribes), and maybe more generally serves as an encouragement to revisit some of our cultural relativism/self-flagellation. But probably not very generalisable. (I note with amusement how unconvincing I find the cognitive process that generated this quote.)

Procrastination is the thief of compound interest.

There can be value to creating witty mottos for our endeavours (e.g. battling akrasia). But such battles aside, this does not feel like it's offering much insight into cognitive processes.

Allow me to express now, once and for all, my deep respect for the work of the experimenter and for his fight to wring significant facts from an inflexible Nature, who says so distinctly "No" and so indistinctly "Yes" to our theories.

If I'm interpreting this correctly, then this can be taken as a quote about the difficulty of locating strong hypotheses. Not particularly epiphanic by Less Wrong standards, but it is clearer than some of the previous examples and does indeed allude to a general protocol.

[A]lmost no innovative programs work, in the sense of reliably demonstrating benefits in excess of costs in replicated RCTs [randomized controlled trials]. Only about 10 percent of new social programs in fields like education, criminology and social welfare demonstrate statistically significant benefits in RCTs. When presented with an intelligent-sounding program endorsed by experts in the topic, our rational Bayesian prior ought to be “It is very likely that this program would fail to demonstrate improvement versus current practice if I tested it.”

In other words, discovering program improvements that really work is extremely hard. We labor in the dark -- scratching and clawing for tiny scraps of causal insight.

Pretty good. General lesson: Without causal insight, we should be suspicious when a string of Promising Solutions fails. Applicable to solutions to problems in one's personal life. Observing an an analogue in tackling mathematical or philosophical problems, this suggests a general attitude to problem-solving of being suspicious of guessing solutions instead of striving for insight.

The use with children of experimental [educational] methods, that is, methods that have not been finally assessed and found effective, might seem difficult to justify. Yet the traditional methods we use in the classroom every day have exactly this characteristic--they are highly experimental in that we know very little about their educational efficacy in comparison with alternative methods. There is widespread cynicism among students and even among practiced teachers about the effectiveness of lecturing or repetitive drill (which we would distinguish from carefully designed practice), yet these methods are in widespread use. Equally troublesome, new "theories" of education are introduced into schools every day (without labeling them as experiments) on the basis of their philosophical or common-sense plausibility but without genuine empirical support. We should make a larger place for responsible experimentation that draws on the available knowledge--it deserves at least as large a place as we now provide for faddish, unsystematic and unassessed informal "experiments" or educational "reforms."

Good. General lesson: Apply reversal tests to complaints against novel approaches, to combat status quo bias.

The general principle of antifragility, it is much better to do things you cannot explain than explain things you cannot do.

Dual of quote before previous. At first I thought I understood this immediately. Then I noticed I was confused and had to remind myself what Taleb's antifragility concept actually is. I feel like it's something to do with doing that which works, regardless of whether we have a good understanding of why it works. I could guess at but am not sure of what the 'explain things you cannot do' part means.

"He keeps saying, you can run, but you can't hide. Since when do we take advice from this guy?"

You got a really good point there, Rick. I mean, if the truth was that we could hide, it's not like he would just give us that information.

Trope deconstruction making a nod to likelihood ratios. Could be taken as a general reminder to be alert to likelihood ratios and incentives to lie. Cool.

Conclusion Out of ten quotes, I would identify two as reinforcing general but basic principles of thought (hypothesis location, likelihood ratios), another that is useful and general (skepticism of Promising Solutions), one which is insightful and general (reversal tests for status quo biases), and one that I wasn't convinced I really grokked but which possibly taught a general lesson (antifragility).

I would call that maybe a score of 2.5 out of 10, in terms of quotes that might actually encourage improvement in general cognitive algorithms. I would therefore suggest something like one of the following:

(1) Be more rigorous in checking that quotes really are rationality quotes before posting them (2) Having two separate threads—one for rationality quotes and one for other quotes (3) Renaming 'Rationality Quotes' to 'Quotes' and just having the one thread. This might seem trivial but it at least decreases the association of non-rationality quotes to the concept of rationality.

I would also suggest that quote posters provide longer quotes to provide context or write the context themselves, and explain the lesson behind the quotes. Some of the above quotes seemed obvious at first, but I mysteriously found that when I tried to formulate them crisply, I found them hard to pin down.

A little bit of How An Algorithm Feels From Inside:

Why is the Monty Hall problem so horribly unintuitive? Why does it feel like there's an equal probability to pick the correct door (1/2+1/2) when actually there's not (1/3+2/3)?

Here are the relevant bits from the Wikipedia article:

Out of 228 subjects in one study, only 13% chose to switch (Granberg and Brown, 1995:713). In her book The Power of Logical Thinking, vos Savant (1996, p. 15) quotes cognitive psychologist Massimo Piattelli-Palmarini as saying "... no other statistical puzzle comes so close to fooling all the people all the time" and "that even Nobel physicists systematically give the wrong answer, and that they insist on it, and they are ready to berate in print those who propose the right answer." Interestingly, pigeons make mistakes and learn from mistakes, and experiments show that they rapidly learn to always switch, unlike humans (Herbranson and Schroeder, 2010).

[...]

Although these issues are mathematically significant, even when controlling for these factors nearly all people still think each of the two unopened doors has an equal probability and conclude switching does not matter (Mueser and Granberg, 1999). This "equal probability" assumption is a deeply rooted intuition (Falk 1992:202). People strongly tend to think probability is evenly distributed across as many unknowns as are present, whether it is or not (Fox and Levav, 2004:637). Indeed, if a player believes that sticking and switching are equally successful and therefore equally often decides to switch as to stay, they will win 50% of the time, reinforcing their original belief. Missing the unequal chances of those two doors, and in not considering that (1/3+2/3) / 2 gives a chance of 50%, similar to "the little green woman" example (Marc C. Steinbach, 2000).

The problem continues to attract the attention of cognitive psychologists. The typical behaviour of the majority, i.e., not switching, may be explained by phenomena known in the psychological literature as: 1) the endowment effect (Kahneman et al., 1991); people tend to overvalue the winning probability of the already chosen – already "owned" – door; 2) the status quo bias (Samuelson and Zeckhauser, 1988); people prefer to stick with the choice of door they have already made. Experimental evidence confirms that these are plausible explanations which do not depend on probability intuition (Morone and Fiore, 2007).

Those bias listed in the last paragraph maybe explain why people choose not to switch the door, but what explains the "equal probability" intuition? Do you have any insight on this?

Does anyone have any advice about understanding implicit communication? I regularly interact with guessers and have difficulty understanding their communication. A fair bit of this has to do with my poor hearing, but I've had issues even on text based communication mediums where I understand every word.

My strategy right now is to request explicit confirmation of my suspicions, e.g., here's a recent online chat I had with a friend (I'm A and they're B):

A: Hey, how have you been?

B: I've been ok

B: working in the lab now

A: Okay. Just to be clear, do you mean that you don't want to be disturbed?

B: yeah

"[W]orking in the lab now" is ambiguous. This friend does sometimes chat online when working in the lab. But, I suspected that perhaps they didn't want to chat, so I asked explicitly.

Requesting explicit confirmation seems to annoy most guessers. I've heard quite a few times that I should "just know" what they mean. Perhaps they think that they have some sort of accurate mental model of others' intentions, but I don't think any of us do. Many guessers have been wrong about my thoughts.

I suspect there probably is no good general strategy other than asking for explicit confirmation. Trying to make guessers be askers is tempting, though probably bound to fail in general.

How do you know when you've had a good idea?

I've found this to actually be difficult to figure out. Sometimes you can google up what you thought. Sometimes checking to see where the idea has been previously stated requires going through papers that may be very very long, or hidden by pay-walls or other barriers on scientific journal sites.

Sometimes it's very hard to google things up. To me, I suppose the standard for "that's a good idea," is if it more clearly explains something I previously observed, or makes it easier or faster for me to do something. But I have no idea whether or not that means it will be interesting for other people.

How do you like to check your ideas?

An experiment with living rationally, by A J Jacobs, who wrote The Year of Living Biblically. I don't know how long he plans to try living rationally.

Posts that have appeared since you last red a page have a pinkish border on them. It's really helpful when dealing with things like open threads and quote threads that you read multiple times. Unfortunately, looking at one of the comments makes it think you read all of them. Clicking the "latest open thread" link just shows one of the comments. This means that, if you see something that looks interesting there, you either have to find the latest open thread yourself, or click the link and have it erase everything about what you have and haven't read.

Can someone make it so looking at one of the comments doesn't reset all of them, or at least put a link to the open thread, instead of just the comments?

To illustrate dead-weight loss in my intro micro class I first take out a dollar bill and give it to a student and then explain that the sum of the wealth of the people in the classroom hasn't changed. Next, I take a second dollar bill and rip it up and throw it in the garbage. My students always laugh nervously as if I've done something scandalous like pulling down my pants. Why?

I've been reading critiques of MIRI, and I was wondering if anyone has responded to this particular critique that basically asks for a detailed analysis of all probabilities someone took into account when deciding that the singularity is going to happen.

(I'd also be interested in responses aimed at Alexander Kruel in general, as he seems to have a lot to say about Lesswrong/Miri.)

Spritz seems like a cool speed reading technique, especially if you have or plan on getting a smart watch. I have no idea how well it works, but I am interested in trying, especially since it does not take a huge training phase. (Click on the phone on that site for a quick demo.)

Does anyone have advice on how to optimize the expectation of a noisy function? The naive approach I've used is to sample the function for a given parameter a decent number of times, average those together, and hope the result is close enough to stand in for the true objective function. This seems really wasteful though.

Most of the algorithms I'm coming (like modelling the objective function with gaussian process regression) would be useful, but are more high-powered than I need. Any simple techniques better than the naive approach? Any recommendations among sophisticated approaches?

Here is a video of someone interviewing people to see if they can guess a pattern by asking whether or not a sequence of 3 numbers satisfies the pattern. (like was mentioned in HPMOR)

Possibly of interest: Help Teach 1000 Kids That Death is Wrong. http://www.indiegogo.com/projects/help-teach-1000-kids-that-death-is-wrong

(have not actually looked in detail, have no opinion yet)

Low priority site enhancement suggestion:

Would it be possible/easy to display the upvotes-to-downvotes ratios as exact fractions rather than rounded percentages? This would make it possible to determine exactly how many votes a comment required without digging through source, which would be nice in quickly determining the difference between a mildly controversial comment and an extremely controversial one.

My eye doctor diagnosed closed-angle glaucoma, and recommends an iridectomy. I think he might be a bit too trigger-happy, so I followed up with another doctor, and she didn't find the glaucoma. She carefully stated that the first diagnosis can still be the correct one, the first was a more complete examination.

Any insights about the pros and cons of iridectomy?

A question I'm not sure how to phrase to Google, and which has so far made Facebook friends think too hard and go back to doing work at work: what is the maximum output bandwidth of a human, in bits/sec? That is, from your mind to the outside world. Sound, movement, blushing, EKG. As long as it's deliberate. What's the most an arbitrarily fast mind running in a human body could achieve?

(gwern pointed me at the Whole Brain Emulation Roadmap; the question of extracting data from an intact brain is covered in Appendix E, but without numbers and mostly with hypothetical technology.)

Someone was asking a while back for meetup descriptions, what you did/ how it went, etc. Figured I'd post some Columbus Rationality videos here. All but the last are from the mega-meetup.

Jesse Galef on Defense Against the Dark Arts: The Ethics and Psychology of Persuasion

Eric on Applications of Models in Everyday Life (it's good, but skip about 10-15 minutes when there's herding-cats-nitpicky audience :P)

Elissa on Effective Altruism

Rita on Cognitive Behavioral Therapy

Don on A Synergy of Eastern and Western Approaches

Gleb on Setting and Achieving Goals

What do you do when you're low on mental energy? I'm a computer science researcher and I have had trouble thinking of anything productive to do when my brain seems to need a break from hard thinking. When I'm really low on mental energy, I go on YouTube, but I feel like that is killing my attention span, and once I go on, I stay too long.

What is the best textbook on datamining? I solemnly swear that upon learning, I intend to use my powers for good.

So, MtGox has declared bankruptcy. Does that make this a good time, or a bad time to invest in Bitcoins? And if a good time, where is the best place to buy them?

I’m basically exactly the kind of person Yvain described here, (minus the passive-aggressive/Machiavellian phase). I notice that that post was sort of a plea for society to behave a different way, but it did not really offer any advice for rectifying the atypical attachment style in the meantime. And I could really use some, because I’ve gotten al-Fulani’d. I’m madly in love in with a woman who does not reciprocate. I’ve actually tried going back on OkCupid to move on, and I literally cannot bring myself to message anyone new, as no one else approaches her either in terms of beauty or in terms of being generally interesting (Despite a tendency to get totally snowed by the halo effect, I’m confident that I would consider her interesting even if she were not so beautiful, though a desire to protect her anonymity prevents me from offering specifics.)

Complicating my situation – when she told me she just wanted to be friends, she actually meant that part. And as she is an awesome person, I don’t want to lose the friendship, which means I’m constantly re-exposed to her and can’t even rely on gradual desensitization. Furthermore, when I asked her if my correcting [failure mode that contributed to her not seeing me in a romantic way] would cause her to reconsider, hoping she’d deliver the coup de grace, she said correcting the failure mode would be a good idea, but she didn’t know whether it would change her feeling about a relationship. This leaves me in the arguably worse situation of having a sliver of hope, however miniscule.

Note that I’m not looking for PUA-type advice here, since a) you would assume from looking at me that I’m an alpha and I have no trouble getting dates, and b) I’m not looking to maximize number of intimate partners.

What I want is advice on a) how not to fall so hard/so fast for (a very small minority of) women, and b)how to break the spell the current one has over me without giving up her friendship. I assume this tendency to rapid, all-consuming affection isn’t an immutable mental trait?

This is one of those times I wish LW allowed explicit politics. SB 1062 in AZ has me craving interesting, rational discussion on the implications of this veto.

One common rationality technique is to put off proposing solutions until you have thought (or discussed) a problem for a while. The goal is to keep yourself from becoming attached to the solutions you propose.

I wonder if the converse approach of "start by proposing lots and lots of solutions, even if they are bad" could be a good idea. In theory, perhaps I could train myself to not be too attached to any given solution I propose, by setting the bar for "proposed solution" to be very low.

In one couples counseling course that I went through, the first step for conflict resolution (after choose a time to discuss and identify the problem) was to together write down at least 10 possible solutions before analyzing any of them. I can perhaps see how this might be more valuable for conflict resolution than for other things, since it gives the other party the sense that you are really trying.

However, it seems plausible to me that even in other contexts, this could be even better than avoiding proposing solutions.

Of course, solution does not have to refer to a proposed action, the same technique could be applied to proposing theories about the cause of some observation.

Thoughts?

I'd like to know where I can go to meet awesome people/ make awesome friends. Occasionally, Yvain will brag about how awesome his social group in the Bay Area was. See here (do read it - its a very cool piece) and I'd like to also have an awesome social circle. As far as I can tell this is a two part problem. The first part is having the requisite social skills to turn strangers into acquaintances and then turn acquaintances into friends. The second part is knowing where to go to find people.

I think that the first part is a solved problem, if you want to learn how to socialize then practice. Which is not to say that it is easy, but its doable. I've heard the suggestion of going to a night club to practice talking to strangers. This is good since people are there to socialize, and I'm sure I could meet all sort of interesting people at one, but I'd like other ideas.

I'd like to know where to go to meet people who I would be likely to get along with. Does anyone have ideas? My list so far

1: Moving to the Bay Area - impractical.

2: Starting a LW meetup - good idea, but it seems like it takes a fair bit of effort.

3: Reaching out into one's extended social circle eg. having a party with your friends and their friends - Probably the most common way people meet new people.

4: Using meetup.com : Not a bad idea

How To Be A Proper Fucking Scientist – A Short Quiz. From Armondikov of RationalWiki, in his "annoyed scientist" persona. A list of real-life Bayesian questions for you to pick holes in the assumptions of^W^W^W^W^W^Wtest yourselves on.

Richard Loosemore (score one for nominative determinism) has a new, well, let's say "paper" which he has, well, let's say "published" here.

His refutation of the usual uFAI scenarios relies solely/mostly on a supposed logical contradiction, namely (to save you a few precious minutes) that a 'CLAI' (a Canonical Logical AI) wouldn't be able to both know about its own fallability/limitations (inevitable in a resource-constrained environment such as reality), and accept the discrepancy between its specified goal system and the creators' actual design intentions. Being superpowerful, the uFAI would notice that it is not following its creator-intended goals but "only" its actually-programmed-in goals*, which, um, wouldn't allow it to continue acting against its creator-intended goals.

So if you were to design a plain ol' garden-variety nuclear weapon intended for gardening purposes ("destroy the weed"), it would go off even if that's not what you actually wanted. However, if you made that weapon super-smart, it would be smart enough to abandon its given goal ("What am I doing with my life?"), consult its creators, and after some deliberation deactivate itself. As such, a sufficiently smart agent would apparently have a "DWIM" (do what the creator means) imperative built-in, which would even supersede its actually given goals -- being sufficiently smart, it would understand that its goals are "wrong" (from some other agent's point of view), and self-modify, or it would not be superintelligent. Like a bizarre version of the argument from evil.

There is no such logical contradiction. Tautologically, an agent is beholden to its own goals, and no other goals. There is no level of capability which magically leads to allowing for fundamental changes to its own goals, on the contrary, the more capable an agent, the more it can take precautions for its goals not to be altered.

If "the goals the superintelligent agent pursues" and "the goals which the creators want the superintelligent agent to pursue, but which are not in fact part of the superintelligent agent's goals" clash, what possible reason would there be for the superintelligent agent to care, or to change itself, changing itself for reasons that squarely come from a category of "goals of other agents (squirrels, programmers, creators, Martians) which are not my goals"? Why, how good of you to ask. There's no such reason for an agent to change, and thus no contradiction.

If someone designed a super-capable killer robot, but by flipping a sign, it came out as a super-capable Gandhi-bot (the horror!), no amount of "but hey look, you're supposed to kill that village" would cause Gandhi-bot to self-modify into a T-800. The bot isn't gonna short-circuit just because someone has goals which aren't its own goals. In particular, there is no capability-level threshold from which on the Gandhi-bot would become a T-800. Instead, at all power levels, it is "content" following its own goals, again, tautologically so.

* In common parlance just called "its goals".