Followup to:  Rationality is Systematized Winning

Yesterday I said:  "Rationality is systematized winning."

"But," you protest, "the reasonable person doesn't always win!"

What do you mean by this?  Do you mean that every week or two, someone who bought a lottery ticket with negative expected value, wins the lottery and becomes much richer than you?  That is not a systematic loss; it is selective reporting by the media.  From a statistical standpoint, lottery winners don't exist—you would never encounter one in your lifetime, if it weren't for the selective reporting.

Even perfectly rational agents can lose.  They just can't know in advance that they'll lose.  They can't expect to underperform any other performable strategy, or they would simply perform it.

"No," you say, "I'm talking about how startup founders strike it rich by believing in themselves and their ideas more strongly than any reasonable person would.  I'm talking about how religious people are happier—"

Ah.  Well, here's the the thing:  An incremental step in the direction of rationality, if the result is still irrational in other ways, does not have to yield incrementally more winning.

The optimality theorems that we have for probability theory and decision theory, are for perfect probability theory and decision theory.  There is no companion theorem which says that, starting from some flawed initial form, every incremental modification of the algorithm that takes the structure closer to the ideal, must yield an incremental improvement in performance.  This has not yet been proven, because it is not, in fact, true.

"So," you say, "what point is there then in striving to be more rational?  We won't reach the perfect ideal.  So we have no guarantee that our steps forward are helping."

Related on OB: Priming and Contamination
Related on LW: When Truth Isn't Enough

When I was a kid, I wanted to be like Mr. Spock on Star Trek.  He was smart, he could kick ass, and he usually saved the day while Kirk was too busy pontificating or womanizing.

And since Spock loved logic, I tried to learn something about it myself.  But by the time I was 13 or 14, grasping the basics of boolean algebra (from borrowed computer science textbooks), and propositional logic (through a game of "Wff'n'Proof" I picked up at a garage sale), I began to get a little dissatisfied with it.

Spock had made it seem like logic was some sort of "formidable" thing, with which you could do all kinds of awesomeness.  But real logic didn't seem to work the same way.

I mean, sure, it was neat that you could apply all these algebraic transforms and dissect things in interesting ways, but none of it seemed to go anywhere.

Logic didn't say, "thou shalt perform this sequence of transformations and thereby produce an Answer".  Instead, it said something more like, "do whatever you want, as long as it's well-formed"...  and left the very real question of what it was you wanted, as an exercise for the logician.

And it was at that point that I realized something that Spock hadn't mentioned (yet): that logic was only the beginning of wisdom, not the end.

Of course, I didn't phrase it exactly that way myself...  but I did see that logic could only be used to check things...  not to generate them.  The ideas to be checked, still had to come from somewhere.

But where?

When I was 17, in college philosophy class, I learned another limitation of logic: or more precisely, of the brains with which we do logic.

Related to: Is That Your True Rejection?

"If you’re interested in being on the right side of disputes, you will refute your opponents’ arguments.  But if you’re interested in producing truth, you will fix your opponents’ arguments for them.  To win, you must fight not only the creature you encounter; you must fight the most horrible thing that can be constructed from its corpse."

   -- Black Belt Bayesian, via Rationality Quotes 13

Yesterday John Maxwell's post wondered how much the average person would do to save ten people from a ruthless tyrant. I remember asking some of my friends a vaguely related question as part of an investigation of the Trolley Problems:

You are a doctor in a small rural hospital. You have ten patients, each of whom is dying for the lack of a separate organ; that is, one person needs a heart transplant, another needs a lung transplant, another needs a kidney transplant, and so on. A traveller walks into the hospital, mentioning how he has no family and no one knows that he's there. All of his organs seem healthy. You realize that by killing this traveller and distributing his organs among your patients, you could save ten lives. Would this be moral or not?

I don't want to discuss the answer to this problem today. I want to discuss the answer one of my friends gave, because I think it illuminates a very interesting kind of defense mechanism that rationalists need to be watching for. My friend said:

It wouldn't be moral. After all, people often reject organs from random donors. The traveller would probably be a genetic mismatch for your patients, and the transplantees would have to spend the rest of their lives on immunosuppressants, only to die within a few years when the drugs failed.

On the one hand, I have to give my friend credit: his answer is biologically accurate, and beyond a doubt the technically correct answer to the question I asked. On the other hand, I don't have to give him very much credit: he completely missed the point and lost a valuable effort to examine the nature of morality.

So I asked him, "In the least convenient possible world, the one where everyone was genetically compatible with everyone else and this objection was invalid, what would you do?"

He mumbled something about counterfactuals and refused to answer. But I learned something very important from him, and that is to always ask this question of myself. Sometimes the least convenient possible world is the only place where I can figure out my true motivations, or which step to take next. I offer three examples:

Followup to:  Moore's Paradox, Doublethink

I don't mean to seem like I'm picking on Kurige, but I think you have to expect a certain amount of questioning if you show up on Less Wrong and say:

One thing I've come to realize that helps to explain the disparity I feel when I talk with most other Christians is the fact that somewhere along the way my world-view took a major shift away from blind faith and landed somewhere in the vicinity of Orwellian double-think.

"If you know it's double-think...

...how can you still believe it?" I helplessly want to say.

Or:

I chose to believe in the existence of God—deliberately and consciously. This decision, however, has absolutely zero effect on the actual existence of God.

If you know your belief isn't correlated to reality, how can you still believe it?

Shouldn't the gut-level realization, "Oh, wait, the sky really isn't green" follow from the realization "My map that says 'the sky is green' has no reason to be correlated with the territory"?

Well... apparently not.

One part of this puzzle may be my explanation of Moore's Paradox ("It's raining, but I don't believe it is")—that people introspectively mistake positive affect attached to a quoted belief, for actual credulity.

But another part of it may just be that—contrary to the indignation I initially wanted to put forward—it's actually quite easy not to make the jump from "The map that reflects the territory would say 'X'" to actually believing "X".  It takes some work to explain the ideas of minds as map-territory correspondence builders, and even then, it may take more work to get the implications on a gut level.

Followup to: Simultaneously Right and Wrong

    "The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents."

          - H.P. Lovecraft, The Call of Cthulhu

There is an old yarn about two skeptics who stayed overnight in a supposedly haunted mansion, just to prove they weren't superstitious. At first, they laughed and joked with each other in the well-lit master bedroom. But around eleven, there was a thunderstorm - hardly a rare event in those parts - and all the lights went off. As it got later and later, the skeptics grew more and more nervous, until finally around midnight, the stairs leading up to their room started to creak. The two of them shot out of there and didn't stop running until they were in their car and driving away.

So the skeptics' emotions overwhelmed their rationality. That happens all the time. Is there any reason to think this story proves anything more interesting than that some skeptics are cowards?

Followup to:  Belief in Self-Deception

Moore's Paradox is the standard term for saying "It's raining outside but I don't believe that it is."  HT to painquale on MetaFilter.

I think I understand Moore's Paradox a bit better now, after reading some of the comments on Less Wrong.  Jimrandomh suggests:

Many people cannot distinguish between levels of indirection. To them, "I believe X" and "X" are the same thing, and therefore, reasons why it is beneficial to believe X are also reasons why X is true.

I don't think this is correct—relatively young children can understand the concept of having a false belief, which requires separate mental buckets for the map and the territory.  But it points in the direction of a similar idea:

Many people may not consciously distinguish between believing something and endorsing it.

After all—"I believe in democracy" means, colloquially, that you endorse the concept of democracy, not that you believe democracy exists.  The word "belief", then, has more than one meaning.  We could be looking at a confused word that causes confused thinking (or maybe it just reflects pre-existing confusion).

So: in the original example, "I believe people are nicer than they are", she came up with some reasons why it would be good to believe people are nice—health benefits and such—and since she now had some warm affect on "believing people are nice", she introspected on this warm affect and concluded, "I believe people are nice".  That is, she mistook the positive affect attached to the quoted belief, as signaling her belief in the proposition.  At the same time, the world itself seemed like people weren't so nice.  So she said, "I believe people are nicer than they are."

Related to: Belief in Belief, Convenient Overconfidence

     "You've no idea of what a poor opinion I have of myself, and how little I deserve it."

      -- W.S. Gilbert

In 1978, Steven Berglas and Edward Jones performed a study on voluntary use of performance inhibiting drugs. They asked subjects to solve certain problems. The control group received simple problems, the experimental group impossible problems. The researchers then told all subjects they'd solved the problems successfully, leaving the controls confident in their own abilities and the experimental group privately aware they'd just made a very lucky guess.

Then they offered the subjects a choice of two drugs to test. One drug supposedly enhanced performance, the other supposedly handicapped it.

There's a cut here in case you want to predict what happened.

Continuation of:  No, Really, I've Deceived Myself
Followup to:  Dark Side Epistemology

I spoke yesterday of my conversation with a nominally Orthodox Jewish woman who vigorously defended the assertion that she believed in God, while seeming not to actually believe in God at all.

While I was questioning her about the benefits that she thought came from believing in God, I introduced the Litany of Tarski—which is actually an infinite family of litanies, a specific example being:

  If the sky is blue
      I desire to believe "the sky is blue"
  If the sky is not blue
      I desire to believe "the sky is not blue".

"This is not my philosophy," she said to me.

"I didn't think it was," I replied to her.  "I'm just asking—assuming that God does not exist, and this is known, then should you still believe in God?"

She hesitated.  She seemed to really be trying to think about it, which surprised me.

"So it's a counterfactual question..." she said slowly.

I thought at the time that she was having difficulty allowing herself to visualize the world where God does not exist, because of her attachment to a God-containing world.

Now, however, I suspect she was having difficulty visualizing a contrast between the way the world would look if God existed or did not exist, because all her thoughts were about her belief in God, but her causal network modelling the world did not contain God as a node.  So she could easily answer "How would the world look different if I didn't believe in God?", but not "How would the world look different if there was no God?"

She didn't answer that question, at the time.  But she did produce a counterexample to the Litany of Tarski:

She said, "I believe that people are nicer than they really are."