My intended next OB post will, in passing, distinguish between moral judgments and factual beliefs.  Several times before, this has sparked a debate about the nature of morality.  (E.g., Believing in Todd.) Such debates often repeat themselves, reinvent the wheel each time, start all over from previous arguments.  To avoid this, I suggest consolidating the debate.  Whenever someone feels tempted to start a debate about the nature of morality in the comments thread of another post, the comment should be made to this post, instead, with an appropriate link to the article commented upon.  Otherwise it does tend to take over discussions like kudzu.  (This isn't the first blog/list where I've seen it happen.)

I'll start the ball rolling with ten points to ponder about the nature of morality...

In Think Like Reality, I put forth the astonishing and controversial proposition that when human intuitions disagree with a fact, we need to either disprove the "fact" in question, or try to reshape the intuition.  (Well, it wouldn't have been so controversial, but like a fool I picked quantum mechanics to illustrate the point.  Never use quantum mechanics as an example of anything.)  Probability theory says that a model which is consistently surprised on the data is probably not a very good model.

Matt Shulman pointed out in personal conversation that, in practice, we may want to be wary of people who don't appear surprised by surprising-seeming data.  Some people affect to be unsurprised because it is a fakeable signal of competence.  Well, a lot of things that good rationalists will do - such as appearing skeptical and appearing to take other people's opinions into account - are also fakeable signals of competence.  But, in practice, Matt's point is still well-taken.

What is true of one apple may not be true of another apple; thus more can be said about a single apple than about all the apples in the world.
       —Twelve Virtues of Rationality 

Within their own professions, people grasp the importance of narrowness; a car mechanic knows the difference between a carburetor and a radiator, and would not think of them both as "car parts".  A hunter-gatherer knows the difference between a lion and a panther.  A janitor does not wipe the floor with window cleaner, even if the bottles look similar to one who has not mastered the art.

Outside their own professions, people often commit the misstep of trying to broaden a word as widely as possible, to cover as much territory as possible.  Is it not more glorious, more wise, more impressive, to talk about all the apples in the world?  How much loftier it must be to explain human thought in general, without being distracted by smaller questions, such as how humans invent techniques for solving a Rubik's Cube.  Indeed, it scarcely seems necessary to consider specific questions at all; isn't a general theory a worthy enough accomplishment on its own?

Imagine looking at your hand, and knowing nothing of cells, nothing of biochemistry, nothing of DNA. You've learned some anatomy from dissection, so you know your hand contains muscles; but you don't know why muscles move instead of lying there like clay. Your hand is just... stuff... and for some reason it moves under your direction. Is this not magic?

"The animal body does not act as a thermodynamic engine ... consciousness teaches every individual that they are, to some extent, subject to the direction of his will. It appears therefore that animated creatures have the power of immediately applying to certain moving particles of matter within their bodies, forces by which the motions of these particles are directed to produce derived mechanical effects... The influence of animal or vegetable life on matter is infinitely beyond the range of any scientific inquiry hitherto entered on. Its power of directing the motions of moving particles, in the demonstrated daily miracle of our human free-will, and in the growth of generation after generation of plants from a single seed, are infinitely different from any possible result of the fortuitous concurrence of atoms... Modern biologists were coming once more to the acceptance of something and that was a vital principle."
        -- Lord Kelvin

This was the theory of vitalism; that the mysterious difference between living matter and non-living matter was explained by an elan vital or vis vitalis.  Elan vital infused living matter and caused it to move as consciously directed. Elan vital participated in chemical transformations which no mere non-living particles could undergo—Wöhler's later synthesis of urea, a component of urine, was a major blow to the vitalistic theory because it showed that mere chemistry could duplicate a product of biology.

Calling "elan vital" an explanation, even a fake explanation like phlogiston, is probably giving it too much credit.  It functioned primarily as a curiosity-stopper.  You said "Why?" and the answer was "Elan vital!"

It is said that parents do all the things they tell their children not to do, which is how they know not to do them.

Long ago, in the unthinkably distant past, I was a devoted Traditional Rationalist, conceiving myself skilled according to that kind, yet I knew not the Way of Bayes.  When the young Eliezer was confronted with a mysterious-seeming question, the precepts of Traditional Rationality did not stop him from devising a Mysterious Answer.  It is, by far, the most embarrassing mistake I made in my life, and I still wince to think of it.

What was my mysterious answer to a mysterious question?  This I will not describe, for it would be a long tale and complicated.  I was young, and a mere Traditional Rationalist who knew not the teachings of Tversky and Kahneman.  I knew about Occam's Razor, but not the conjunction fallacy.  I thought I could get away with thinking complicated thoughts myself, in the literary style of the complicated thoughts I read in science books, not realizing that correct complexity is only possible when every step is pinned down overwhelmingly.  Today, one of the chief pieces of advice I give to aspiring young rationalists is "Do not attempt long chains of reasoning or complicated plans."

Nothing more than this need be said:  Even after I invented my "answer", the phenomenon was still a mystery unto me, and possessed the same quality of wondrous impenetrability that it had at the start.

Make no mistake, that younger Eliezer was not stupid.  All the errors of which the young Eliezer was guilty, are still being made today by respected scientists in respected journals.  It would have taken a subtler skill to protect him, than ever he was taught as a Traditional Rationalist.

Indeed, the young Eliezer diligently and painstakingly followed the injunctions of Traditional Rationality in the course of going astray.

Continuation of:  My Wild and Reckless Youth

Once upon a time, in my wild and reckless youth, when I knew not the Way of Bayes, I gave a Mysterious Answer to a mysterious-seeming question.  Many failures occurred in sequence, but one mistake stands out as most critical:  My younger self did not realize that solving a mystery should make it feel less confusing.  I was trying to explain a Mysterious Phenomenon—which to me meant providing a cause for it, fitting it into an integrated model of reality.  Why should this make the phenomenon less Mysterious, when that is its nature?  I was trying to explain the Mysterious Phenomenon, not render it (by some impossible alchemy) into a mundane phenomenon, a phenomenon that wouldn't even call out for an unusual explanation in the first place.

As a Traditional Rationalist, I knew the historical tales of astrologers and astronomy, of alchemists and chemistry, of vitalists and biology.  But the Mysterious Phenomenon was not like this.  It was something new, something stranger, something more difficult, something that ordinary science had failed to explain for centuries—

- as if stars and matter and life had not been mysteries for hundreds of years and thousands of years, from the dawn of human thought right up until science finally solved them—

We learn about astronomy and chemistry and biology in school, and it seems to us that these matters have always been the proper realm of science, that they have never been mysterious.  When science dares to challenge a new Great Puzzle, the children of that generation are skeptical, for they have never seen science explain something that feels mysterious to them.  Science is only good for explaining scientific subjects, like stars and matter and life.

Followup to: Semantic Stopsigns, Mysterious Answers to Mysterious Questions

As our tribe wanders through the grasslands, searching for fruit trees and prey, it happens every now and then that water pours down from the sky.

"Why does water sometimes fall from the sky?" I ask the bearded wise man of our tribe.

He thinks for a moment, this question having never occurred to him before, and then says, "From time to time, the sky spirits battle, and when they do, their blood drips from the sky."

"Where do the sky spirits come from?" I ask.

His voice drops to a whisper.  "From the before time.  From the long long ago."

When it rains, and you don't know why, you have several options.  First, you could simply not ask why—not follow up on the question, or never think of the question in the first place.  This is the Ignore command, which the bearded wise man originally selected.  Second, you could try to devise some sort of explanation, the Explain command, as the bearded man did in response to your first question.  Third, you could enjoy the sensation of mysteriousness—the Worship command.

Followup to: Semantic Stopsigns, Mysterious Answers to Mysterious Questions, Say Not 'Complexity'

Imagine that I, in full view of live television cameras, raised my hands and chanted abracadabra and caused a brilliant light to be born, flaring in empty space beyond my outstretched hands.  Imagine that I committed this act of blatant, unmistakeable sorcery under the full supervision of James Randi and all skeptical armies. Most people, I think, would be fairly curious as to what was going on.

But now suppose instead that I don't go on television.  I do not wish to share the power, nor the truth behind it. I want to keep my sorcery secret.  And yet I also want to cast my spells whenever and wherever I please. I want to cast my brilliant flare of light so that I can read a book on the train—without anyone becoming curious.  Is there a spell that stops curiosity?

Yes indeed!  Whenever anyone asks "How did you do that?", I just say "Science!"

It's not a real explanation, so much as a curiosity-stopper. It doesn't tell you whether the light will brighten or fade, change color in hue or saturation, and it certainly doesn't tell you how to make a similar light yourself. You don't actually know anything more than you knew before I said the magic word. But you turn away, satisfied that nothing unusual is going on.

Better yet, the same trick works with a standard light switch.

Continuation of:  What is Evidence?

Light leaves the Sun and strikes your shoelaces and bounces off; some photons enter the pupils of your eyes and strike your retina; the energy of the photons triggers neural impulses; the neural impulses are transmitted to the visual-processing areas of the brain; and there the optical information is processed and reconstructed into a 3D model that is recognized as an untied shoelace; and so you believe that your shoelaces are untied.

Here is the secret of deliberate rationality—this whole entanglement process is not magic, and you can understand it.  You can understand how you see your shoelaces.  You can think about which sort of thinking processes will create beliefs which mirror reality, and which thinking processes will not.

Mice can see, but they can't understand seeing.  You can understand seeing, and because of that, you can do things which mice cannot do.  Take a moment to marvel at this, for it is indeed marvelous.

In "What is Evidence?", I wrote:

This is why rationalists put such a heavy premium on the paradoxical-seeming claim that a belief is only really worthwhile if you could, in principle, be persuaded to believe otherwise.  If your retina ended up in the same state regardless of what light entered it, you would be blind...  Hence the phrase, "blind faith".  If what you believe doesn't depend on what you see, you've been blinded as effectively as by poking out your eyeballs.

Cihan Baran replied:

I can not conceive of a situation that would make 2+2 = 4 false. Perhaps for that reason, my belief in 2+2=4 is unconditional.