Eliezer wrote a post warning against unrealistically confident estimates, in which he argued that you can't be 99.99% sure that 53 is prime. Chris Hallquist replied with a post arguing that you can.

That particular case is tricky. There have been many independent calculations of the first hundred prime numbers. 53 is a small enough number that I think someone would notice if Wikipedia included it erroneously. But can you be 99.99% confident that 1159 is a prime? You found it in one particular source. Can you trust that source? It's large enough that no one would notice if it were wrong. You could try to verify it, but if I write a Perl or C++ program, I can't even be 99.9% sure that the compiler or interpreter will interpret it correctly, let alone that the program is correct.

Rather than argue over the number of nines to use for a specific case, I want to emphasize the the importance of not assigning things probability zero or one. Here's a real case where approximating 99.9999% confidence as 100% had disastrous consequences.

In grade school, I read a series of books titled Sideways Stories from Wayside School by Louis Sachar, who you may know as the author of the novel Holes which was made into a movie in 2003. The series included two books of math problems, Sideways Arithmetic from Wayside School and More Sideways Arithmetic from Wayside School, the latter of which included the following problem (paraphrased):

The students have Mrs. Jewl's class have been given the privilege of voting on the height of the school's new flagpole. She has each of them write down what they think would be the best hight for the flagpole. The votes are distributed as follows:

• 1 student votes for 6 feet.
• 1 student votes for 10 feet.
• 7 students vote for 25 feet.
• 1 student votes for 30 feet.
• 2 students vote for 50 feet.
• 2 students vote for 60 feet.
• 1 student votes for 65 feet.
• 3 students vote for 75 feet.
• 1 student votes for 80 feet, 6 inches.
• 4 students vote for 85 feet.
• 1 student votes for 91 feet.
• 5 students vote for 100 feet.

At first, Mrs. Jewls declares 25 feet the winning answer, but one of the students who voted for 100 feet convinces her there should be a runoff between 25 feet and 100 feet. In the runoff, each student votes for the height closest to their original answer. But after that round of voting, one of the students who voted for 85 feet wants their turn, so 85 feet goes up against the winner of the previous round of voting, and the students vote the same way, with each student voting for the height closest to their original answer. Then the same thing happens again with the 50 foot option. And so on, with each number, again and again, "very much like a game of tether ball."

Question: if this process continues until it settles on an answer that can't be beaten by any other answer, how tall will the new flagpole be?

Answer (rot13'd): fvkgl-svir srrg, orpnhfr gung'f gur zrqvna inyhr bs gur bevtvany frg bs ibgrf. Naq abj lbh xabj gur fgbel bs zl svefg rapbhagre jvgu gur zrqvna ibgre gurberz.

Why am I telling you this? There's a minor reason and a major reason. The minor reason is that this shows it is possible to explain little-known academic concepts, at least certain ones, in a way that grade schoolers will understand. It's a data point that fits nicely with what Eliezer has written about how to explain things. The major reason, though, is that a month ago I finished my systematic read-through of the sequences and while I generally agree that they're awesome (perhaps moreso than most people; I didn't see the problem with the metaethics sequence), I thought the mini-discussion of political parties and voting was on reflection weak and indicative of a broader nerd failure mode.

TLDR (courtesy of lavalamp):

1. Politicians probably conform to the median voter's views.
2. Most voters are not the median, so most people usually dislike the winning politicians.
3. But people dislike the politicians for different reasons.
4. Nerds should avoid giving advice that boils down to "behave optimally". Instead, analyze the reasons for the current failure to behave optimally and give more targeted advice.

Related to: Utilons vs. Hedons, Would Your Real Preferences Please Stand Up

And I don't mean that question in the semantic "but what is happiness?" sense, or in the deep philosophical "but can anyone not facing struggle and adversity truly be happy?" sense. I mean it in the totally literal sense. Are wireheads having fun?

They look like they are. People and animals connected to wireheading devices get upset when the wireheading is taken away and will do anything to get it back. And it's electricity shot directly into the reward center of the brain. What's not to like?

Only now neuroscientists are starting to recognize a difference between "reward" and "pleasure", or call it "wanting" and "liking". The two are usually closely correlated. You want something, you get it, then you feel happy. The simple principle behind our entire consumer culture. But do neuroscience and our own experience really support that?

Followup to:  Fake Explanations, Guessing the Teacher's Password

Phlogiston was the 18 century's answer to the Elemental Fire of the Greek alchemists.  Ignite wood, and let it burn.  What is the orangey-bright "fire" stuff?  Why does the wood transform into ash?  To both questions, the 18th-century chemists answered, "phlogiston".

...and that was it, you see, that was their answer:  "Phlogiston."

Phlogiston escaped from burning substances as visible fire.  As the phlogiston escaped, the burning substances lost phlogiston and so became ash, the "true material".  Flames in enclosed containers went out because the air became saturated with phlogiston, and so could not hold any more.  Charcoal left little residue upon burning because it was nearly pure phlogiston.

Of course, one didn't use phlogiston theory to predict the outcome of a chemical transformation.  You looked at the result first, then you used phlogiston theory to explain it.  It's not that phlogiston theorists predicted a flame would extinguish in a closed container; rather they lit a flame in a container, watched it go out, and then said, "The air must have become saturated with phlogiston."  You couldn't even use phlogiston theory to say what you ought not to see; it could explain everything.

This was an earlier age of science.  For a long time, no one realized there was a problem.  Fake explanations don't feel fake.  That's what makes them dangerous.

Followup to:  Circular Altruism.  Response to:  Knowing your argumentative limitations, OR "one [rationalist's] modus ponens is another's modus tollens."

(Still no Internet access.  Hopefully they manage to repair the DSL today.)

I haven't said much about metaethics - the nature of morality - because that has a forward dependency on a discussion of the Mind Projection Fallacy that I haven't gotten to yet.  I used to be very confused about metaethics.  After my confusion finally cleared up, I did a postmortem on my previous thoughts.  I found that my object-level moral reasoning had been valuable and my meta-level moral reasoning had been worse than useless.  And this appears to be a general syndrome - people do much better when discussing whether torture is good or bad than when they discuss the meaning of "good" and "bad".  Thus, I deem it prudent to keep moral discussions on the object level wherever I possibly can.

Occasionally people object to any discussion of morality on the grounds that morality doesn't exist, and in lieu of jumping over the forward dependency to explain that "exist" is not the right term to use here, I generally say, "But what do you do anyway?" and take the discussion back down to the object level.

Paul Gowder, though, has pointed out that both the idea of choosing a googolplex dust specks in a googolplex eyes over 50 years of torture for one person, and the idea of "utilitarianism", depend on "intuition".  He says I've argued that the two are not compatible, but charges me with failing to argue for the utilitarian intuitions that I appeal to.

[PDF of this article updated Aug. 23, 2011]

[skip to preface]

Whenever I write a new article for Less Wrong, I'm pulled in two opposite directions.

One force pulls me toward writing short, exciting posts with lots of brain candy and just one main point. Eliezer has done that kind of thing very well many times: see Making Beliefs Pay Rent, Hindsight Devalues Science, Probability is in the Mind, Taboo Your Words, Mind Projection Fallacy, Guessing the Teacher's Password, Hold Off on Proposing Solutions, Applause Lights, Dissolving the Question, and many more.

Another force pulls me toward writing long, factually dense posts that fill in as many of the pieces of a particular argument in one fell swoop as possible. This is largely because I want to write about the cutting edge of human knowledge but I keep realizing that the inferential gap is larger than I had anticipated, and I want to fill in that inferential gap quickly so I can get to the cutting edge.

For example, I had to draw on dozens of Eliezer's posts just to say I was heading toward my metaethics sequence. I've also published 21 new posts (many of them quite long and heavily researched) written specifically because I need to refer to them in my metaethics sequence.¹ I tried to make these posts interesting and useful on their own, but my primary motivation for writing them was that I need them for my metaethics sequence.

And now I've written only four posts² in my metaethics sequence and already the inferential gap to my next post in that sequence is huge again. :(

So I'd like to try an experiment. I won't do it often, but I want to try it at least once. Instead of writing 20 more short posts between now and the next post in my metaethics sequence, I'll attempt to fill in a big chunk of the inferential gap to my next metaethics post in one fell swoop by writing a long tutorial post (a la Eliezer's tutorials on Bayes' Theorem and technical explanation).³

So if you're not up for a 20-page tutorial on human motivation, this post isn't for you, but I hope you're glad I bothered to write it for the sake of others. If you are in the mood for a 20-page tutorial on human motivation, please proceed.

A few years back, my great-grandmother died, in her nineties, after a long, slow, and cruel disintegration.  I never knew her as a person, but in my distant childhood, she cooked for her family; I remember her gefilte fish, and her face, and that she was kind to me.  At her funeral, my grand-uncle, who had taken care of her for years, spoke:  He said, choking back tears, that God had called back his mother piece by piece: her memory, and her speech, and then finally her smile; and that when God finally took her smile, he knew it wouldn't be long before she died, because it meant that she was almost entirely gone.

I heard this and was puzzled, because it was an unthinkably horrible thing to happen to anyone, and therefore I would not have expected my grand-uncle to attribute it to God.  Usually, a Jew would somehow just-not-think-about the logical implication that God had permitted a tragedy.  According to Jewish theology, God continually sustains the universe and chooses every event in it; but ordinarily, drawing logical implications from this belief is reserved for happier occasions.  By saying "God did it!" only when you've been blessed with a baby girl, and just-not-thinking "God did it!" for miscarriages and stillbirths and crib deaths, you can build up quite a lopsided picture of your God's benevolent personality.

Hence I was surprised to hear my grand-uncle attributing the slow disintegration of his mother to a deliberate, strategically planned act of God. It violated the rules of religious self-deception as I understood them.

In L. Sprague de Camp's fantasy story The Incomplete Enchanter (which set the mold for the many imitations that followed), the hero, Harold Shea, is transported from our own universe into the universe of Norse mythology.  This world is based on magic rather than technology; so naturally, when Our Hero tries to light a fire with a match brought along from Earth, the match fails to strike.

I realize it was only a fantasy story, but... how do I put this...

No.

Followup to:  The Genetic Fallacy, Where Recursive Justification Hits Bottom

Yesterday I said that—because humans are not perfect Bayesians—the genetic fallacy is not entirely a fallacy; when new suspicion is cast on one of your fundamental sources, you really should doubt all the branches and leaves of that root, even if they seem to have accumulated new evidence in the meanwhile.

This is one of the most difficult techniques of rationality (on which I will separately post, one of these days).  Descartes, setting out to "doubt, insofar as possible, all things", ended up trying to prove the existence of God—which, if he wasn't a secret atheist trying to avoid getting burned at the stake, is pretty pathetic.  It is hard to doubt an idea to which we are deeply attached; our mind naturally reaches for cached thoughts and rehearsed arguments.

But today's post concerns a different kind of difficulty—the case where the doubt is so deep, of a source so fundamental, that you can't make a true fresh beginning.

Case in point:  Remember when, in the The Matrix, Morpheus told Neo that the machines were harvesting the body heat of humans for energy, and liquefying the dead to feed to babies?  I suppose you thought something like, "Hey!  That violates the second law of thermodynamics."

I remember a poster that hung on the wall of my seventh grade classroom. It went like this:

Watch your thoughts, for they become words.
Watch your words, for they become actions.
Watch your actions, for they become habits.
Watch your habits, for they become your character.
Watch your character, for it becomes your destiny.

It was as a competitive swimmer that these words were the most meaningful to me. Most sports are ultimately about the practice, about repeating an action over and over and over again, so that actions become habits and habits become character. The fleeting thought that I really hate getting up at 5:00 am for swim practice is just that: a fleeting thought. But if I justified it with words, speaking it aloud to my parents or siblings or friends, it became a fact that others knew about me, much realer than just a wispy thought. The action of forgetting-on-purpose to set my alarm, or faking sick, was a logical next step. And one missed practice might not be huge, in the long run, but it led easily to a habit of missing practice, say, once a week. A year of this, and I would start to think of myself as the kind of person who missed practice once a week, because after all, isn’t it silly of anyone to expect a twelve-year-old to get up at 5:00 three times a week? And that attitude could very easily have led, over a couple of years, to quitting the team.