This claim seems incorrect:
[mean squared error] wouldn't penalize a credence of 99.999% on wrong answers enough to strongly dissuade unjustified absolute certainties.
Since the scoring rule is proper, the penalty for incorrectly being 99.999% confident is exactly high enough that you wouldn't want to say you are if you're merely 99.998% confident.
Unless you mean something different?
Hey Alexei, I know it's been a while since this game was developed, but I've discovered it recently and I want to make a better one. More question types, community-contributed and voted questions, categories, profiles; cross-platform web app, themes, better UI etc.
It's of course going to be open-source.
Would you like to help with math model for questions and evaluating, or maybe with development?
galatyuk.ilya AT gmail
Hey rationality friends, I just made this FAQ for the credence calibration game. So if you have people you'd like to introduce to it --- for example, to get them used to thinking of belief strengths as probabilities --- now is a good time :)