Exercise material: List of suggested conclusions to rationalize. I'm not quite sure what the prerequisites here would be; one of my own childhood epiphanies was finding that I could argue reasons why there ought to be a teapot in the asteroid belt, and thinking to myself, "Well, I better not do that, then." I suspect the primary desideratum would be more that the topic offers plenty of opportunity to come up with clever rationalizations, rather than that the topic offers motivation to come up with clever rationalizations.
Exercise A: Pick a conclusion from the list. Come up with a clever argument why it is true. Notice what it feels like to do this. Then don't do it.
(This is distinct from noticing the feeling of being tempted to rationalize, a separate subskill.)
Exercise B: In the middle of coming up with clever arguments why something is true, stop and chain into the Litany of Tarski, or some other remedy. Variant: Say out loud "Ew!" or "Oops!" during the stop part.
Exercise C: For a conclusion on the list, argue that it is true. Then "Ask whether, not why" - try to figure out whether it is true. Notice the difference between these two processes. Requires a question whose answer is less than immediately obvious, but which someone can find info about by searching their memory and/or the Internet.
Hmmm this just made me think that debate clubs deliberately teach the opposite of this skill.
Recent brainstorming sessions at SIAI (with participants including Anna, Carl, Jasen, Divia, Will, Amy Willey, and Andrew Critch) have started to produce lists of rationality skills that we could potentially try to teach (at Rationality Boot Camp, at Less Wrong meetups, or similar venues). We've also been trying to break those skills down to the 5-second level (step 2) and come up with ideas for exercises that might teach them (step 3) although we haven't actually composed those exercises yet (step 4, where the actual work takes place).
The bulk of this post will mainly go into the comments, which I'll try to keep to the following format: A top-level comment is a major or minor skill to teach; upvote this comment if you think this skill should get priority in teaching. Sub-level comments describe 5-second subskills that go into this skill, and then third-level comments are ideas for exercises which could potentially train that 5-second skill. If anyone actually went to the work of composing a specific exercise people could run through, that would go to the fourth-level of commenting, I guess. For some major practicable arts with a known standard learning format like "Improv" or "Acting", I'll put the exercise at the top and guesses at which skills it might teach below. (And any plain old replies can go at any level.)
I probably won't be able to get to all of what we brainstormed today, so here's a PNG of the Freemind map that I generated during our session.