A couple of times I've accidentally clicked twice in rapid succession on the "Continue" button, and the second click has been interpreted as an answer to the next question. (I think it happened to be a correct answer both times.)
I suggest disabling the buttons -- or at least the "B" ones -- for about 100ms after displaying a new question.
Hey rationality friends, I just made this FAQ for the credence calibration game. So if you have people you'd like to introduce to it --- for example, to get them used to thinking of belief strengths as probabilities --- now is a good time :)