Web app idea: I'm posting this comment immediately and without editing so I don't forget the idea before I get a chance to write it down/work it out more, as I have to leave my computer soon.

1. Display a short passage that illustrates something irrational that people do or think, with instructions for the reader to enter into a text box the first or most important thing that came to mind and then press a "ready" button.
2. Ready button reveals a question of the form, "Were your thoughts similar to any of the following?" with a list of questions/remarks you would hope a rationalist would (or wouldn't) ask/make.
3. Yes/No buttons save text box and button answers, clear the text box and question fields and replace the passage with a new one.

No priming by reading questions before passages. Writing their thoughts before seeing the question will hopefully keep people honest. Saving text box with answers allows answer auditing. Each passage's irrationality may be more or less obvious depending on a person's background. Same with desired/undesired thinking examples with questions (that's what we're measuring with this though, isn't it?).

Positive example question: Yes = +1, No = 0 Negative example question: Yes = -1, No = 0

With even split between positive/negative example questions, rationalists should score = 1/2 # of questions asked. More questions answered = more confidence in estimate. Wider range of topics addressed in questions = more confidence in estimate.

Edited to add: I created a storyboard for the app's testing process here and have started a list of example passages with desired/undesired responses here.