I have been looking for a good calibration test for a while! Thank you.
Here's a little critique: The questions are becoming repetitive after playing it for not too long. I have noticed that a few of the questions are wrong, as in it says that I am wrong and docks points but I am right according to the data it sites (I think some of these had to do with the dark knight).
Also, many of the questions are US centric and many of the questions are about sports or popular music or other things that I would not expect the average person interested in calibration to knowledgeable about or interested in. If you are trying to make it take off these might be justified however, these could be good decisions.
Hey rationality friends, I just made this FAQ for the credence calibration game. So if you have people you'd like to introduce to it --- for example, to get them used to thinking of belief strengths as probabilities --- now is a good time :)