Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.


30 Post author: Eliezer_Yudkowsky 06 October 2007 07:24PM

I remember the exact moment when I began my journey as a rationalist.

It was not while reading Surely You're Joking, Mr. Feynman or any existing work upon rationality; for these I simply accepted as obvious.  The journey begins when you see a great flaw in your existing art, and discover a drive to improve, to create new skills beyond the helpful but inadequate ones you found in books.

In the last moments of my first life, I was fifteen years old, and rehearsing a pleasantly self-righteous memory of a time when I was much younger.  My memories this far back are vague; I have a mental image, but I don't remember how old I was exactly.  I think I was six or seven, and that the original event happened during summer camp.

What happened originally was that a camp counselor, a teenage male, got us much younger boys to form a line, and proposed the following game: the boy at the end of the line would crawl through our legs, and we would spank him as he went past, and then it would be the turn of the next eight-year-old boy at the end of the line.  (Maybe it's just that I've lost my youthful innocence, but I can't help but wonder...)  I refused to play this game, and was told to go sit in the corner. 

This memory—of refusing to spank and be spanked—came to symbolize to me that even at this very early age I had refused to take joy in hurting others.  That I would not purchase a spank on another's butt, at the price of a spank on my own; would not pay in hurt for the opportunity to inflict hurt.  I had refused to play a negative-sum game.

And then, at the age of fifteen, I suddenly realized that it wasn't true.  I hadn't refused out of a principled stand against negative-sum games.  I found out about the Prisoner's Dilemma pretty early in life, but not at the age of seven.  I'd refused simply because I didn't want to get hurt, and standing in the corner was an acceptable price to pay for not getting hurt.

More importantly, I realized that I had always known this—that the real memory had always been lurking in a corner of my mind, my mental eye glancing at it for a fraction of a second and then looking away.

In my very first step along the Way, I caught the feeling—generalized over the subjective experience—and said, "So that's what it feels like to shove an unwanted truth into the corner of my mind!  Now I'm going to notice every time I do that, and clean out all my corners!"

This discipline I named singlethink, after Orwell's doublethink.  In doublethink, you forget, and then forget you have forgotten.  In singlethink, you notice you are forgetting, and then you remember.  You hold only a single non-contradictory thought in your mind at once.

"Singlethink" was the first new rationalist skill I created, which I had not read about in books.  I doubt that it is original in the sense of academic priority, but this is thankfully not required.

Oh, and my fifteen-year-old self liked to name things.

The terrifying depths of the confirmation bias go on and on.  Not forever, for the brain is of finite complexity, but long enough that it feels like forever.  You keep on discovering (or reading about) new mechanisms by which your brain shoves things out of the way.

But my young self swept out quite a few corners with that first broom.


Part of the Against Doublethink subsequence of How To Actually Change Your Mind

Next post: "Doublethink (Choosing to be Biased)"

Previous post: "The Sacred Mundane" (in previous subsequence)

Comments (24)

Sort By: Old
Comment author: Bob_Knaus 06 October 2007 11:28:26PM 15 points [-]

Jeez, what's next? "Optimism Bias and My First Kiss"?

As an avid reader, I have to say I'm enjoying the personal revelations. They have much in common with certain varieties of religious exuberance. In fact the whole sequence has the feel of an Olde Tyme Rationalist Revival.

But butt smacking recollections from summer camp? Surely there has to be a line somewhere.

Comment author: Eliezer_Yudkowsky 06 October 2007 11:32:22PM 20 points [-]

I know. I can't help it. That was my first step into advanced rationality. I mean, what was I supposed to say? It was an important point and I wanted to get it across.

The truth doesn't always save a person. It especially doesn't always save a person from embarrassment.

Comment author: JulianMorrison 07 October 2007 12:05:58AM 9 points [-]

This sort of thing intrigues me. It underlines for me that our "technology of the mind" is still very young and unstructured.

Imagine what would happen if this one skill were explicitly taught in schools?

Comment author: DaFranker 03 August 2012 01:46:49PM 4 points [-]

Out of curiosity, if you're still reading LessWrong, has this question resurfaced often since then? Do you think (or anyone think, since this is by no means a closed personal question) progress has been made on this "front"? Is the principle applicable, and is it realistic to have rationality be explicitly taught in public education? Is CFAR exactly what you had in mind back then?

Comment author: MarsColony_in10years 03 April 2015 02:11:11AM 1 point [-]

I have been thinking this ever since I started the sequences. Another idea would be to try to produce children's books, and get them distributed as widely as possible.

Comment author: Constant2 07 October 2007 12:28:52AM 6 points [-]

I prefer to see examples like this. There are lots of good reasons for grounding these discussions in the concrete.

Comment author: Richard_Hollerith 07 October 2007 01:08:09AM 5 points [-]

An exercise I adopted when I was a child that I did not read anywhere is to recall or reconstruct the thought that led to my current thought, then the thought before that, etc. By examining many such transitions, I discovered some generalizations about the unconscious function that takes thought N to thought N+1.

Comment author: [deleted] 18 February 2012 09:21:01PM 1 point [-]

I do the same, too. (But I had never thought of it as an exercise, just curiosity about “Why the deuce am I thinking about this?”.) I think there was an xkcd comic about that, too, but I can't find it now.

Comment author: Hank 07 October 2007 02:48:33AM 1 point [-]

Did you ever actually assert that your reasoning was out of principle, only to suddenly remember the experience through an unrelated pathway and thus the remember the real reason of your refusal later on, contradicting that initial assertion, or was it simply that you had never actually considered your reason of refusal until you were 15, and that it had only *felt* like you had been making the assumption that you had refused out of principle when you had never asserted that as truth- you had just never considered your reason, and you knew when you were 15 that there was reason to refuse- because it was a negative-sum game.

Comment author: Constant2 07 October 2007 07:59:06AM 4 points [-]

So, Eliezer. You're interested in the Singularity. And the Singularity will come about by means of self-improving artificial intelligence. If I understand correctly. Meanwhile, you're also interested in self-improvement of the human wetware - hence your contributions to this website. There's an obvious similarity between these two interests, possibly even an overlap.

Took me a while to notice it.

Comment author: Recovering_irrationalist 07 October 2007 12:36:36PM 2 points [-]


By examining many such transitions, I discovered some generalizations about the unconscious function that takes thought N to thought N+1.

I'm very interested, please elaborate!


You started noticing the value of beating specific biases at age 15, but apparently didn't get the importance of heuristics and biases in general till 2003. Why did that reflective generalisation take so long, when in hindsight it's so vital for your chosen path? That's not meant as criticism - I've done this too. I want to know how to fix it.

Incentive to think about this: doing so might lead you to notice something vital for safe seed AI.

Comment author: Eliezer_Yudkowsky 07 October 2007 02:28:43PM 2 points [-]

RI, the problem was simply the size of the planet and its scientific literature. I just didn't run into the field of heuristics and biases until that late. I realized it was important the first time I read a mention-in-passing on a webpage - though I didn't realize how much there was to know and how little of it I knew; Emil Gilliam finally mailed me a copy of "Judgment Under Uncertainty".

At the age of fifteen, I realized the importance of defeating biases in general, I just didn't know there was a field that studied them. I knew the importance of rationality in general, but I'd never heard the word "Bayesian". It's a frighteningly big world out there - not like a hunter-gatherer tribe, where you can presume you know everything anyone else knows.

Comment author: Richard_Hollerith 07 October 2007 07:42:26PM 0 points [-]

-- and since most writings in psychology are worthless, it is easy to give up on the whole field before one discovers worthwhile writings like "Judgement Under Uncertainty" and "The Moral Animal".

RI, a large part of my motivatation was simply to practice a mental skill: it is a delightful feeling to improve drastically one's ability to observe one's own deliberations. Three decades and a severe bump on the head separate my teenage years from today, and today I am almost completely unable to do this exercise.

BTW, it is my guess that the exercises Eliezer and I describe will confer most of their benefits on exercisers who are still teenagers.

RI, to answer your question: the function that takes thought N into thought N+1 is complex enough that I did not learn anything that could be put into neat sentences, nor do I retain any declarative memories of what I learned except that the deliberation proceeded in a much more "predictable-in-retrospect" manner when I thought about some themes than when I thought about others. E.g., I remember that thinking about my mom produced very opaque chain of thoughts.

The practice Eliezer describes strikes me as of greater potential benefit than the one I describe, but perhaps the one I describe can be accomplished by a greater fraction of teenagers reading these words. Very few individuals are blessed with the delightful hardware that the teenage Eliezer had available for such exercises.

Comment author: Recovering_irrationalist 08 October 2007 12:17:06AM 0 points [-]

Thanks for your replies,

and since most writings in psychology are worthless, it is easy to give up on the whole field

Fair point. But if it annoys us, think how the rational minds in the field must feel!

...it is a delightful feeling... ...did not learn anything that could be put into neat sentences... ...much more "predictable-in-retrospect"...

Are you sure the model that your mind was using to report its own thought process corresponded to what was really going on under the bonnet? I've barely scratched the surface yet in studying bias and rationality (the 2 books you recommended are queuing on my shelf) but a recurring theme seems to be that introspection about one's own thought processes is often demonstrably unreliable, even among highly intelligent subjects working within their fields of expertise and reporting very high confidence in their introspections.


Maybe I'm just biased towards cross-fertilization between disciplines. I was just wondering about other vital but relatively obscure knowledge being missed for similar reasons, and if there might be a way to attack the problem. Anyhow, please thank Emil for us, a lot of people are getting stronger thanks to her little gift! :-)

Comment author: Recovering_irrationalist 08 October 2007 12:02:49PM 0 points [-]

...her little gift...

That's what I get for posting at 1am. Apologies to Emil, who I see is an SIAI donor and therefore one of my favorite people :-)

Comment author: Nathan3 08 October 2007 02:09:16PM 0 points [-]

That's a horrible game. Did you have a name for the game as well? Spank-to-spank?

Comment author: TGGP4 08 October 2007 09:42:54PM 2 points [-]

I remember playing a game in which we would take turns punching each other in the stomach as hard as we could while trying not to flinch, but I viewed it as positive sum. Even though you get punched it's still a lot of fun.

Comment author: Eliezer_Yudkowsky 12 October 2009 07:33:16PM 3 points [-]

Glad to know it wasn't just my own summer camp which was that deranged.

Might've been fun for you, but was it fun for everyone who was pressured into playing? I'm also slightly skeptical of your after-the-fact judgment that the initiation ritual was fun. But not too skeptical, for different people are different.

Comment author: buybuydandavis 29 October 2011 05:40:17AM 3 points [-]

The ones for whom it was not fun were the ones in most need of the game.

It's natural to be afraid of pain. It's your body's signal for damage. But pain in that kind of controlled environment, where you know you are fundamentally safe, and no damage will occur, gives you a chance to get over your fear of pain, which in most cases is much more painful than the pain itself.

There are few things that a child will face that is more painful than the fear of pain. I think it's healthy for a child to be gently led out of that fear. When I was young, I was too afraid, and wish that someone had helped me to get past it.

Comment author: christopherj 13 December 2013 05:16:14AM *  0 points [-]

Glad to know it wasn't just my own summer camp which was that deranged.

Far from it. I suspect it's a fairly universal game, either to signal "I am tough", or practice for handling pain or fear.

The version of this game popular in my country involves taking turns kicking a soccer ball at each other from a given distance, usually while the target is facing a wall. Whoever kicks the ball is up next, even if they miss. I assume it is a positive sum game, because kids would play this game of their own volition, though that could be because they don't want to refuse when invited. It was a rather uncommon game, relative to proper games.

I've also seen a game called "mercy" where the fingers of both parties are interlaced and bent backwards until someone gives up, the game called "chicken" involving driving cars at each other (which I've heard a lot about but never seen played), the crushing handshake, truth or dare. However, the most intriguing one is the drinking contest.

Comment author: Laura 09 October 2007 03:58:38AM 3 points [-]

Interesting. As a child I thought I could remember everything, so lying was easy, it's own memory of having lied, and thus distinct from reality. It was only much later that I realized it was even possible for the two to become horribly confused in one's own mind. But did you lie and forget the lie? Or did you just incompletely remember and add the part about negative sums as an explanation of the behavior later on? I think the "sweeping" of our minds' corners is also prone to give us false memories. We think, "well, why did I do that? Oh yeah! Now I *remember*..." You might have just been lucky to stumble upon what you now believe to be the real memory, but introspection, especially of the past isn't always so accurate.

Comment author: Richard_Hollerith 10 October 2007 05:10:05PM 0 points [-]

Are you sure the model that your mind was using to report its own thought process corresponded to what was really going on under the bonnet?

I was very confident that I could reliably retain or reconstruct the last five or so actual thoughts of my conscious deliberation -- and the actual ordering or sequencing of those thoughts. Although recalling or reconstructing a thought sometimes required a few seconds of effort or patience, the reconstruction sure felt like a reliable observation rather than an interpretation or the updating of a model or the use of a learned model. As for extracting generalizations or regularities from those observations, like I said, I can recall making only one (declarative) generalization, but I have no reason to believe that I could not bring the same regularity-noticing, causality-extracting powers to bear on those observations that I could to any observations made by my senses. (When observing a very complex process, the regularity-noticing consists mostly of implicit learning rather than explicit, declarative learning -- or so it seems to me.)

Comment author: Endovior 29 March 2011 08:10:38PM 4 points [-]

Interesting. Now I'm thinking about my own journey.

Having been raised Christian... and in a very evangelical, pentecostal tradition no less... I wasn't exactly encouraged to think like this early on... great swathes of things were simply to be taken on faith, and doubt simply referred over to the appropriately doctrinal section of apologetics. It did not help that both of my parents hold bachelor's degrees in Christian theology.

Nonetheless... they couldn't entirely shield me from the world. The start of my journey, so far as I can recall, came while studying physics... which I did before high school, incidentally, as I was always a voracious reader. I read about things like Einstein's Theory of Relativity, and Heisenberg's Uncertainty Principle... and after thinking about the implications of them, I realized that I was confused.

"Hang on, this can't be right. God is omniscient, and thus has knowledge of everything, everywhere in the universe, irregardless of such pettiness as the speed of light being the maximum speed for transfer of information. The tools of Science may not be able to determine both the exact position and momentum of a particle, but God must know!"

Of course, my initial reaction was wrong. But I'd spotted a problem in my worldview, and additional study of science seemed to reinforce my nagging doubts. I didn't have a word to explain it, then... indeed, I didn't have the words to fully articulate the problem until I read Atlas Shrugged in college, and thereby learned of the law of non-contradiction. But it was seeing that contradiction that first got me thinking in terms of "either my religious beliefs are true, or a large body of experimentally verified science is true, but not both"... and having realized that, it left me open to changing my mind on the subject later on. Which, fortunately, I did.

Comment author: buybuydandavis 29 October 2011 05:46:19AM *  2 points [-]

The first little heuristic trick toward mental health. It seems to me that children actually used to be taught such things, and weren't left to bring themselves up like wolf boys.

For some reason, this article had my mind wandering to "My Favorite Things". Turns out, there was a reason. Think Julie Andrews, and sing along:

When the dog bites, when the bee stings When I'm feeling sad, I simply remember my favorite things and then I don't feeel so bad!

Weren't children's stories full of such things, once upon a time? Every story had some lesson to be learned. I don't have kids, but my impression is that every story these days aims at imparting a proper attitude, not a useful skill.