Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Scott Aaronson's "On Self-Delusion and Bounded Rationality"

16 Post author: cousin_it 18 August 2009 07:17PM

Poignant short story about truth-seeking that I just found. Quote:

"No," interjected an internal voice. "You need to prove that your dad will appear by a direct argument from the length of your nails, one that does not invoke your subsisting in a dream state as an intermediate step."

"Nonsense," retorted another voice. "That we find ourselves in a dream state was never assumed; rather, it follows so straightforwardly from the long-nail counterfactual that the derivation could be done, I think, even in an extremely weak system of inference."

The full thing reads like a flash tour of OB/LW, except it was written in 2001.

Comments (48)

Comment author: dclayh 18 August 2009 11:09:35PM 7 points [-]

Two cars race toward each other on an empty freeway; the first to swerve is the chicken. How should you play if you want to preserve both your status and your life? The answer is clear: in full view of your opponent, rip out your car's steering wheel, blindfold yourself, down a bottle of Jack Daniels, scream. If you can persuade your opponent that you're incapable of making the decision to swerve, then he has to swerve. In other words: the stupider, more ignorant, more irrational you can prove you are, the better the chance you have of winning.

I'd like to remind everyone that at one point this strategy was the de facto foreign policy of the United States.

Comment author: cousin_it 19 August 2009 12:26:05AM 5 points [-]

Obligatory link to our Schelling thread.

Comment author: Alicorn 18 August 2009 08:06:44PM 12 points [-]

Around the middle of the story, the protagonist started to remind me of myself in elementary school. I had not yet learned the word "atheist", and so when classmates asked after my religion, I had nothing to tell them. This was uncomfortable, so I decided to make something up. It involved unicorns, and thankfully, I was never able to quite push myself to the point where I believed any of it.

Comment author: SilasBarta 21 August 2009 07:33:51PM 3 points [-]

LOL! Awesome :-)

Comment author: CronoDAS 19 August 2009 02:49:16AM 4 points [-]

If only Eric had had a better response to this:

"What you're saying is tantamount to saying that you want to fuck me. So why shouldn't I react with revulsion precisely as though you'd said the latter?"

then maybe tragedy would have been averted?

Comment author: Alicorn 19 August 2009 02:55:04AM *  12 points [-]

Agreed. Eric displayed so little ability to react to oddness. Perhaps, "Why would you react with revulsion if I expressed the desire to fuck you? If it's because of cultural expectation or because you would interpret such an expression as an indication that I'm generally abrupt and crass, then the analogy fails to hold."

Comment author: Psychohistorian 19 August 2009 04:53:17AM *  10 points [-]

This is an instance of Hollywood-style rationality: it sounds like she's being "rational," but she takes a single implication and just stops thinking. This is doubly true because she then cites strong evidence that pretty much all of the guys she's around want to fuck her, so merely looking at her is a pretty good sign of this fact, so it seems likely that his asking her out is probably indicative of something extra. If the guy in question had been sharp enough to beat her at her own game, I wouldn't be surprised if she'd then wanted to go out with him.

But this hits a broader point: people with the mental sharpness needed to break a fairly intelligent Hollywood Rationalist are few and far between. Her misery seems to stem more from a lack of like-minded peers than it does from anything about the nature of her position. If she'd found a very sharp rationalist who'd taken interest in her and given her approval, she'd probably have been happy that way. This may say more about the importance of finding a like-minded peer group than about being hyper-rational.

Comment author: Furcas 19 August 2009 05:27:09AM *  9 points [-]

I have to say I'm surprised by the amount of praise this story is getting.

The main character seems convinced that the difficulty she experiences in interacting pleasantly with members of the opposite sex and possibly starting a relationship with someone less rational than she is, is due to her inability to delude herself, or even to compartmentalize.

But it's not. It's due to her inability to shut up once in a while. Instead of working on changing her entire psyche, couldn't she have simply made an effort to, you know, control the way she behaves?

Epistemic rationality has nothing to do with extreme honesty towards other individuals, or with showing contempt for irrationalists, or even with feeling contempt for them. The greatest epistemic rationalist on Earth could have a happy relationship with a Young Earth Creationist; all s/he'd have to do is either refrain from criticism, or be very polite and gentle about it.

Also, I wasn't very impressed with the classification of Richard Dawkins (and those like him) as a "a Type-1-and-higher retard". What he is is a good Type-1-and-higher thinker who cares about the truth and therefore to whom avoiding self-deception is advantageous.

Comment author: Simon_Jester 21 August 2009 08:15:25PM 15 points [-]

I think you're misreading the story. It's not an argument in favor of irrationality, it's a horror story. The catch is that it's a good horror story, directed at the rationalist community. Like most good horror stories, it plays off a specific fear of its audience.

You may be immune to the lingering dread created by looking at all those foolish happy people around you and wondering if maybe you are the one doing something wrong. Or the fear that even if you act as rationally as you can, you could still box yourself into a trap you won't be able to think your way back out of. But quite a few of your peers are not so immune. I know I'm not, and that story managed to scare me pretty effectively.

The protagonist isn't an ideal rationalist, and the story isn't trying to assert that this is what the ideal rationalist does. Instead, the protagonist is an adolescent proto-rationalist, of a type many of us are familiar with, with her social instincts sucking her into a trap that a lot of us can understand well enough to dread.

And so there's a reason she thinks and acts like a Hollywood stereotype of an intelligent person is that, especially when they're just barely at the age of being able to really think at all. Where do you think Hollywood got the idea for the stereotype in the first place?

I submit that the reason so many of the average people think intelligent people act that way is because they lose social contact with the geniuses in high school, which is when they do think and act like that.

For a lot of the smartest people, being socially functional is a learned skill that comes late and not easily.

Comment author: Nominull 21 August 2009 08:32:09PM 4 points [-]

Agreed. I went in expecting a parable against rationality, and about halfway through I realized I was reading existential horror (the best kind).

The writing isn't great and the points are made hamhandedly, but there is the core of a good story here.

Comment author: Furcas 21 August 2009 09:05:06PM *  1 point [-]

I've upvoted this comment, but I disagree.

What should make this an effective horror story, as you put it, is that it's based on the very real possibility that there are people whose brains are wired in such a way that they can't be happy and rational at the same time. In order to more effectively 'scare' the reader, the author attempts to convince us that this is more than a possibility by making an argument by fictional example, the example being the main character.

My beef with the story is that this example is way too unlikely to be convincing as an argument (and therefore scary as a horror story). If there are people who can't possibly be rational and happy, I'm pretty sure it's not because they're incapable of keeping their tongues under control in order to start a relationship on the right foot.

Comment author: Simon_Jester 22 August 2009 01:18:48AM 0 points [-]

I dunno. I mean, a lot of horror stories that are famous for being good talk about stuff that can never be and should never be, but that nonetheless (in-story) is. I think it's that sense of a comforting belief about the world being violated that makes a good horror story, even if the prior probability of that belief being wrong is low.

Comment author: cousin_it 19 August 2009 10:06:00AM 4 points [-]

Least convenient possible world for your objection: the protagonist couldn't change her liability to shut up, but could change her rationality.

Comment author: Furcas 19 August 2009 03:36:57PM *  4 points [-]

That's possible, but so unlikely that it strains credulity (haha). Like I said in my reply to Kaj Sotala, this story is trying to make a point about some humans who live in the real world. Have you ever heard of someone for whom changing from a 'hyper-rationalist' to a deluded fool is easier than to learn to keep one's mouth shut every now and then?

Comment author: wedrifid 27 August 2009 03:50:50PM 2 points [-]

Have you ever heard of someone for whom changing from a 'hyper-rationalist' to a deluded fool is easier than to learn to keep one's mouth shut every now and then?

Yes.

It is also curious to observe that in humans extroversion is strongly correlated with conformism.

Comment author: rhollerith_dot_com 27 August 2009 06:10:57PM *  2 points [-]

It is also curious to observe that in humans extroversion is strongly correlated with conformism.

Has a study found such a correlation, wedrifid, or are you going by personal observations here?

Comment author: wedrifid 27 August 2009 07:39:34PM 0 points [-]

Yes, more than one study has found such a correlation. My personal observations agree but not to the extent that I would rule out the fairly obvious potential biasses in my perception even if I had noticed it.

(No, I don't have the reference on hand.)

Comment author: rhollerith_dot_com 27 August 2009 09:45:34PM *  0 points [-]

Thanks! Any idea how the studies you mention measured conformism? I'm guessing they administered questions like "The conventional wisdom is usually right."

Comment author: Kaj_Sotala 19 August 2009 09:45:26AM 4 points [-]

The main character seems convinced that the difficulty she experiences... is due to her inability... but it's not.

Who said that the characters in good stories must never be mistaken?

Comment author: Furcas 19 August 2009 03:32:56PM *  2 points [-]

Sure, but this isn't just a story, it's a story that tries to make a point about the negative consequences of rationality for some humans in the real world. Her belief isn't merely her belief, it's also a step in an argument.

Comment author: Kaj_Sotala 19 August 2009 08:21:16PM 0 points [-]

I suspect you may be reading too much into it: I thought the "negative consequences of rationality" were more for the sake of pure comedy than for making any kind of a point.

Of course, you may also be right and I might be the one who's reading too little into it.

Comment author: SforSingularity 19 August 2009 10:35:05PM -2 points [-]

The greatest epistemic rationalist on Earth could have a happy relationship with a Young Earth Creationist;

false in general, false as a statistical statement too.

Comment author: eirenicon 20 August 2009 12:32:01AM 2 points [-]

The greatest epistemic rationalist on Earth is still made out of meat.

Comment author: Furcas 19 August 2009 11:19:06PM 2 points [-]

I was making a point about human thought processes, not human desires. I agree that it's unlikely that the greatest epistemic rationalist would want to have a relationship with a YEC, but if s/he did want to, s/he could.

Comment author: Eliezer_Yudkowsky 19 August 2009 11:38:31PM 6 points [-]

If I were otherwise unattached, I would totally have a relationship with a YEC, if she was from a world which had actually been created 6000 years ago. Otherwise no.

Comment author: Alicorn 20 August 2009 12:02:35AM 8 points [-]

What if she was just from a world where lots of evidence pointed to it having been created 6,000 years ago, but it was really created last Thursday?

Comment author: Eliezer_Yudkowsky 20 August 2009 12:17:16AM 14 points [-]

Is she hot?

...okay, that was a bit out of character, but I think that at that point in the thread I basically had no choice but to say that.

Comment author: Alicorn 19 August 2009 10:38:39PM *  1 point [-]

Unless you have actually tracked down and interviewed the greatest epistemic rationalist on earth, how do you know? Maybe (s)he is very tolerant of such things. (When does intolerance win on a personal scale?)

Comment author: SforSingularity 19 August 2009 11:11:54PM *  1 point [-]

how do you know?

This sounds like scientism.

Because I have experience with good rationalists, and the kind of people they have relationships with, and I am a bayesian so I can assign degrees of belief to propositions that I haven't tested directly. In this case, it seems reasonable that similar people have similar relationship-behaviors, and so my existing knowledge is relevant.

Rather like "how do you know that the fastest dog in the world can't outrun a formula one car?" - I know this with high certainty because I believe that similar animals behave in similar ways.

Comment author: whowhowho 27 February 2013 03:25:34PM 0 points [-]

If it is false they could not. What would prevent them?

Comment author: Psychohistorian 18 August 2009 11:48:07PM *  2 points [-]

Very good story. The change in grammar and writing style may be condescending towards a certain group of people, and the character(s) are perhaps a little too much like caricatures, but it's overall quite interesting and well-written.

It also highlights a rather massive potential disconnect between rationality and utilitarianism. It seems hard to argue that she's not hedonically happier at the end, but I still find it rather depressing. This isn't exactly news, but I feel the issue gets swept under the carpet rather often in these parts.

Comment author: cousin_it 19 August 2009 12:35:41AM *  -2 points [-]

Hedonically happier, huh? A Roissy quote seems appropriate:

Happiness is not granted from on high nor is it a wispy feeling that randomly alights on your mood. Happiness is constructed. It is the direct result of actions taken that further your genes’ goals of survival and reproduction.

I would've amended it to say "likely to have furthered your genes' goals in the ancestral environment", but the original formulation fits Aaronson's story like a glove.

Comment author: Psychohistorian 19 August 2009 02:27:26AM *  4 points [-]

Happiness is constructed. It is the direct result of actions taken that further your genes’ goals of survival and reproduction.

Blatantly obvious counterexamples include forced reproduction and rape. Less blatantly obvious counterexamples include masturbation and bacon double-cheeseburgers, though these don't apply to your amended version. The fact that we have the capacity to experience hedonic pleasure because it was adaptive does not mean we experience hedonic pleasure as a direct result of acting to increase our inclusive genetic fitness.

It may very well fit the story like a glove, but that doesn't make it any less egregious of a misunderstanding of evolutionary biology. She's hedonically happier, and she seems to have increased her odds of reproducing; direct causality does not seem present. If her amazing new boyfriend is sterile, it isn't really going to change the equation that much.

Comment author: cousin_it 19 August 2009 10:02:19AM *  5 points [-]

Being raped wasn't likely to further your genes' goals in the ancestral environment, because it decreases your chances to reproduce consensually with a better mate and maybe have him provide resources for the kids.

Comment author: Alicorn 19 August 2009 03:35:05PM 3 points [-]

You're ignoring spousal rape, where a better mate is already only going to be had through the spouse's death or hazardous subterfuge, and resource support for the kids can indeed be expected.

Comment author: SilasBarta 19 August 2009 03:52:47PM 0 points [-]

I'm skeptical of Roissy's claim, but I'm not sure your examples are as obvious or convincing as you seem to think.

Unless I'm extremely atypical (which is possible), it is incorrect to say that superstimulant foods and masturbation bring happiness. They bring contentment, which is different and less satisfying. To the extent that this is a definitional dispute, the happiness they bring is far weaker and far more evanescent compared to e.g. feeling like part of a group, feeling like you've "made a difference", etc.

Next, regarding forced reproduction, if it had to be forced, it was a kind of reproduction that the victim's evolved architecture did not judge to be genetically optimal.

Rape is a bit trickier, for a number of reasons, but it's clear why it would have questionable, not stricktly good impact on genes. As cousin_it mentioned, someone who rapes you is unlikely to stick around to provide resources for the kids. And like with forced reproduction, the fact that it had to be forced is evidence your body doesn't regard it as furthering your genes.

Now, to play devil's advocate, there is the Sexy Son Hypothesis, WHICH I DON'T ENDORSE, which says that the actual father's willingness to commit is irrelevant because you can just rook a productive male into thinking the baby's his. So being raped means you get a son (if a boy) who can successfully rape other women, who then rook another man into raising that son, etc. But again, against the clear disadvantages of being raped, this at best proves, under Roissy's theory, why the woman would feel conflicted, not happy.

Comment author: pjeby 19 August 2009 04:14:52PM *  2 points [-]

Now, to play devil's advocate, there is the Sexy Son Hypothesis, WHICH I DON'T ENDORSE

Endorse meaning, "consider true", or endorse meaning, "consider morally good"? I could understand the latter (so few things about evolution can be considered "good", after all), but don't get the former. Do you think it's untrue?

Comment author: SilasBarta 19 August 2009 04:22:55PM *  0 points [-]

Enough people who read that post won't be able to tell the difference.

You can figure it out from there, I think.

Comment author: wedrifid 27 August 2009 04:08:33PM 0 points [-]

Enough people who read that post won't be able to tell the difference.

The only clear implication I can take from that is that you believe the people in this environment are poor at distinguishing between statements of fact and moral assertions when it comes to potentially sensitive topics.

Oh. Hang on. Took me a while but I can figure it out from there after all. I've reversed my downvote. Reasonable answer.

Comment author: pjeby 19 August 2009 06:14:38PM 2 points [-]

Next, regarding forced reproduction, if it had to be forced, it was a kind of reproduction that the victim's evolved architecture did not judge to be genetically optimal.

By the way, this isn't necessarily the case. Depending on the species, it could also be the case that force is part of the selection process for determining a mate's suitability. Evolution isn't a kind and loving god, alas. (Some species' mating rituals involve the death of one or both partners, for example.)

Comment author: SilasBarta 19 August 2009 07:48:23PM 0 points [-]

I mean "forced", as I think PH did, in the sense of "the being that was forced did not want to do it". The cases of beetles that get their heads bitten off after mating with a female wouldn't be a relevant example, since, in the appropriate sense, the male seems to want that.

But then, if resistance is part of some test, than PH's example wouldn't be relevant anyway.

Comment author: Psychohistorian 19 August 2009 09:41:03PM *  0 points [-]

My response is to the Roissy quote specifically, which says, "direct result of actions taken that further your genes' goals" NOT "would have furthered your genes' goals in the ancestral environment." Thus, in the case of forced reproduction, if you were locked in a small room and given just enough food to survive, but then given the opportunity to reproduce with a fertile partner every single time that you were capable of causing conception, you would (I'd hope) not be happy, even though you would produce many, many more times the surviving offspring (if you're male, literally thousands of times the surviving offspring). It seems almost trivially obvious that objectively maximizing inclusive genetic fitness will not make you happy.

The amendment "tend to maximize... in the ancestral environment" changes the equation completely. First, it's very prone to "just-so" stories; one can invent a decent explanation as to why practically any action increases genetic fitness (i.e. celibacy is for aunt/unclehood). Second, because the cause is tremendously indirect, it isn't realistic to say that the cause of someone's increased happiness is the fact that they are (or would be in the AE) more genetically fit, as outlined in the EY post I linked. There's a great deal of truth to it, but it's both indirect and tremendously imprecise. And, more generally, if someone asks "How can I be happier?" the answer is seldom, "Well, think about what would maximize your inclusive genetic fitness in the ancestral environment, and then do that."

On a related note, my problem with a lot of Roissy's reasoning is that he views people in terms of rather binary "objective" Platonic ideals of "male" and "female" and does not account for either innate or learned differences, instead dismissing such as trivial, which is quite problematic, because a great diversity of preferences actually exist.

Edit: Also, wikipedia's phrasing of the sexy son hypothesis contradicts what I've read elsewhere. The idea is that if females generally prefer trait X, then females should choose males with trait X, even if trait X is completely superficial, because then their sons will be more successful with future females. That doesn't make non-superficial trait Y irrelevant, and there is no basis in reason nor a citation claiming it should. The hypothesis is simply an explanation of how an expensive superficial trait can become dominant; it has nothing (explicitly) to do with cuckoldry or anything like that.

Comment author: dclayh 18 August 2009 09:16:03PM *  3 points [-]

A quite good if depressing story. But he ruins the brilliant Flowers for Algernon parallel by pointing it out explicitly.

Comment author: ajayjetti 18 August 2009 10:04:34PM 1 point [-]

Few quotes from the article:

"Bananas were created by God for human enjoyment, for why else would they come in such convenient cases?"

"...Am I to hope that, in the hereafter, a rationalist God will reward me for having the intellectual integrity not to believe in Him?".....

...Right, but if the only reason it works is that you believe it works, then how can it work if you know it only works because you believe it works?"...

..."Richard Dawkins, the biologist, was once asked about a study claiming that the devout live longer on average than atheists. He replied that, even if that were so, he'd rather know the truth about where he came from and die early than live longer under a fantasy."

. .... "In other words: the stupider, more ignorant, more irrational you can prove you are, the better the chance you have of winning".

When Douglas Yates wrote that "people who are sensible about love are incapable of it," he might have added a footnote: "the Darwinian explanation for this fact arises from certain paradoxes of rationality in games played by agents known to each other to have bounded computational capacity."

Comment author: Eliezer_Yudkowsky 18 August 2009 09:08:09PM *  0 points [-]

Oldie but goodie. Maybe emphasize the link a bit harder, took me a moment to find it.

Comment author: cousin_it 19 August 2009 12:24:53AM 0 points [-]

Done.