CarlShulman comments on Making your explicit reasoning trustworthy - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (93)
Interesting! But I do think it's harder than we imagine to maintain that perfect firewall between arguments you read and arguments you believe (or at least absorb into your decisions). Cases where you're genuinely uncertain about the truth are probably more salient than cases like Scientology on this front.
Well, yeah. Scientology is sort of the Godwin example of dangerous infectious memes. But I've found the lessons most useful in dealing with lesser ones, and it taught me superlative skills in how to inspect memes and logical results in a sandbox.
Perhaps these have gone to the point where I've recompartmentalised and need to aggressively decompartmentalise again. Anna Salamon's original post is IMO entirely too dismissive of the dangers of decompartmentalisation in the Phil Goetz post, which is about people who accidentally decompartmentalise memetic toxic waste and come to the startling realisation they need to bomb academics or kill the infidel or whatever. But you always think it'll never happen to you. And this is false, because you're running on unreliable hardware with all manner of exploits and biases, and being able to enumerate them doesn't grant you immunity. And there are predators out there, evolved to eat people who think it'll never happen to them.
My own example: I signed up for a multi-level marketing company, which only cost me a year of my life and most of my friends. I should detail precisely how I reasoned myself into it. It was all very logical. The process of reasoning oneself into the mouth of a highly evolved predator tends to be. The cautions my friends and family gave me were all heuristic. This was before I studied Scientology in detail, which would I suspect have given me some immunity.
I should write a post on the subject (see my recent comments) except Anna's post covers quite a lot of it.
I would be interested in reading this, and especially about what caused the initial vulnerability.