Compartmentalization in epistemic and instrumental rationality

77 AnnaSalamon 17 September 2010 07:02AM

Related to: Humans are not automatically strategic, The mystery of the haunted rationalist, Striving to accept, Taking ideas seriously

I argue that many techniques for epistemic rationality, as taught on LW, amount to techniques for reducing compartmentalization.  I argue further that when these same techniques are extended to a larger portion of the mind, they boost instrumental, as well as epistemic, rationality.

Imagine trying to design an intelligent mind.

One problem you’d face is designing its goal.  

Every time you designed a goal-indicator, the mind would increase action patterns that hit that indicator[1].  Amongst these reinforced actions would be “wireheading patterns” that fooled the indicator but did not hit your intended goal.  For example, if your creature gains reward from internal indicators of status, it will increase those indicators -- including by such methods as surrounding itself with people who agree with it, or convincing itself that it understood important matters others had missed.  It would be hard-wired to act as though “believing makes it so”. 

A second problem you’d face is propagating evidence.  Whenever your creature encounters some new evidence E, you’ll want it to update its model of  “events like E”.  But how do you tell which events are “like E”? The soup of hypotheses, intuition-fragments, and other pieces of world-model is too large, and its processing too limited, to update each belief after each piece of evidence.  Even absent wireheading-driven tendencies to keep rewarding beliefs isolated from threatening evidence, you’ll probably have trouble with accidental compartmentalization (where the creature doesn’t update relevant beliefs simply because your heuristics for what to update were imperfect).

Evolution, AFAICT, faced just these problems.  The result is a familiar set of rationality gaps:

continue reading »

Striving to Accept

33 Eliezer_Yudkowsky 09 March 2009 11:29PM

Reply toThe Mystery of the Haunted Rationalist
Followup toDon't Believe You'll Self-Deceive

Should a rationalist ever find themselves trying hard to believe something?

You may be tempted to answer "No", because "trying to believe" sounds so stereotypical of Dark Side Epistemology.  You may be tempted to reply, "Surely, if you have to try hard to believe something, it isn't worth believing."

But Yvain tells us that - even though he knows damn well, on one level, that spirits and other supernatural things are not to be found in the causal closure we name "reality" - and even though he'd bet $100 against $10,000 that an examination would find no spirits in a haunted house - he's pretty sure he's still scared of haunted houses.

Maybe it's okay for Yvain to try a little harder to accept that there are no ghosts, since he already knows that there are no ghosts?

In my very early childhood I was lucky enough to read a book from the children's section of a branch library, called "The Mystery of Something Hill" or something.  In which one of the characters says, roughly:  "There are two ways to believe in ghosts.  One way is to fully believe in ghosts, to look for them and talk about them.  But the other way is to half-believe - to make fun of the idea of ghosts, and talk scornfully of ghosts; but to break into a cold sweat when you hear a bump in the night, or be afraid to enter a graveyard."

continue reading »