pjeby comments on Raising the Sanity Waterline - Less Wrong

112 Post author: Eliezer_Yudkowsky 12 March 2009 04:28AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (207)

You are viewing a single comment's thread. Show more comments above.

Comment author: RobinHanson 12 March 2009 01:04:15PM 10 points [-]

I suspect you are right; the issue isn't that these people haven't "learned" relevant abstractions or tools. They just don't have enough incentives to apply those tools in these context. I'm not sure you "teach" incentives, so I'm not sure there is anything you can teach which will achieve the goal stated. So I'd ask the question: how can we give people incentives to apply their tools to cases like religion?

Comment author: pjeby 12 March 2009 03:32:28PM 10 points [-]

It's not incentive either. I have plenty of incentive, and so do my students. It's simply that we don't notice our beliefs as beliefs, if they're already in our heads. (As opposed to the situation when vetting input that's proposed as a new belief.)

Since we don't have any kind of built-in function for listing ALL the beliefs involved in a given decision, we are often unaware of the key beliefs that are keeping us stuck in a particular area. We sit there listing all the "beliefs" we can think of, while the single most critical belief in that area isn't registering as a "belief" at all; it just fades in as part of our background assumptions. To us, it's something like "water is wet" -- sure it's a belief, but how could it possibly be relevant to our problem?

Usually, an irrational fear associated with something like, "but how will I pay the bills?" masquerades as simple, factual logic. But the underlying emotional belief is usually something more like, "If I don't pay the bills, then I'm an irresponsible person and no-one will love me." The underlying belief is invisible because we don't look underneath the "logic" to find the emotion hiding underneath.

Unfortunately, all reasoning is motivated reasoning, which means that to find your irrational beliefs in a given area, you have to first dig up a nontrivial number of rationalizations... knowing that the rationalization you're looking for is probably something you specifically created to prevent you from thinking about the motivation involved in the first place! (After all, revealing to others that you think you're irresponsible isn't good genetic fitness... and if you know, that makes it more likely you'll unintentionally reveal it.)

A simple tool, by the way, for digging up the motivation behind seemingly "factual" statements and beliefs is to ask, "And what's bad about that?" or "And what's good about that?".... usually followed by, "And what does that say/mean about YOU?" You pretty quickly discover that nearly everything in the universe revolves around you. ;-)