Just to add more data to my experience: I broke the technique into steps when I tried it. The steps did seem helpful. But maybe someone who used the technique more successfully could give us better lines of approach. The steps I used (I wrote out answers to each step, since writing or speaking aloud to myself makes it easier to follow out difficult lines of thought):
Step 1: Pick a question.
Step 2: Find a reason to care about having actually an accurate answer. Find something to protect that hinges on believing whatever it is that’s accurate about this question, whether or not that accurate answer turns out to be my current belief.
Step 3: Notice any reasons I might want to stick to my current belief, even if that belief turns out to be untrue. See if they in fact outweigh the reasons to want the actual answer.
Step 4: Create doubt or curiosity: Find my current belief’s weakest points, or the points I am most afraid to consider. Go through a list of outside people I respect, and ask what points they might balk at in my beliefs. Ask if my belief has anything in common with past errors I’ve made, or if an uncharitable stranger might think so. Ask if anything I'm saying to myself makes me feel squicky. Brainstorm. Ask what the space of alternatives might look like.
Step 5: Actually do the Crisis of Faith technique. [Except that I didn't get to this step.]
Since there's been much questioning of late over "What good is advanced rationality in the real world?", I'd like to remind everyone that it isn't all about post-doctoral-level reductionism.
In particular, as a technique that seems like it ought to be useful in the real world, I exhibit the highly advanced, difficult, multi-component Crisis of Faith aka Reacting To The Damn Evidence aka Actually Changing Your Mind.
Scanning through this post and the list of sub-posts at the bottom (EDIT: copied to below the fold) should certainly qualify it as "extreme rationality" or "advanced rationality" or "x-rationality" or "Bayescraft" or whatever you want to distinguish from "traditional rationality as passed down from Richard Feynman".
An actual sit-down-for-an-hour Crisis of Faith might be something you'd only use once or twice in every year or two, but on important occasions. And the components are often things that you could practice day in and day out, also to positive effect.
I think this is the strongest foot that I could put forward for "real-world" uses of my essays. (Anyone care to nominate an alternative?)
Below the fold, I copy and paste the list of components from the original post, so that we have them at hand: