CarlShulman comments on Making your explicit reasoning trustworthy - Less Wrong

82 Post author: AnnaSalamon 29 October 2010 12:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (93)

You are viewing a single comment's thread. Show more comments above.

Comment author: alexflint 31 October 2010 09:10:06PM 1 point [-]

Interesting! But I do think it's harder than we imagine to maintain that perfect firewall between arguments you read and arguments you believe (or at least absorb into your decisions). Cases where you're genuinely uncertain about the truth are probably more salient than cases like Scientology on this front.

Comment author: David_Gerard 01 November 2010 12:37:16AM *  15 points [-]

Well, yeah. Scientology is sort of the Godwin example of dangerous infectious memes. But I've found the lessons most useful in dealing with lesser ones, and it taught me superlative skills in how to inspect memes and logical results in a sandbox.

Perhaps these have gone to the point where I've recompartmentalised and need to aggressively decompartmentalise again. Anna Salamon's original post is IMO entirely too dismissive of the dangers of decompartmentalisation in the Phil Goetz post, which is about people who accidentally decompartmentalise memetic toxic waste and come to the startling realisation they need to bomb academics or kill the infidel or whatever. But you always think it'll never happen to you. And this is false, because you're running on unreliable hardware with all manner of exploits and biases, and being able to enumerate them doesn't grant you immunity. And there are predators out there, evolved to eat people who think it'll never happen to them.

My own example: I signed up for a multi-level marketing company, which only cost me a year of my life and most of my friends. I should detail precisely how I reasoned myself into it. It was all very logical. The process of reasoning oneself into the mouth of a highly evolved predator tends to be. The cautions my friends and family gave me were all heuristic. This was before I studied Scientology in detail, which would I suspect have given me some immunity.

I should write a post on the subject (see my recent comments) except Anna's post covers quite a lot of it.

Comment author: CarlShulman 02 November 2010 02:19:12PM 3 points [-]

I would be interested in reading this, and especially about what caused the initial vulnerability.