The fun idea for scoring guessing games is to wait till the result is known, and then give each contestant points according to the logarithm of the probability that they had previously given the to outcome that actually occurred.
(Hmm, all the scores are negative; some are more negative than others. Perhaps the score keeper needs to give log n - log p points if he wants the numbers to be human-friendly. (The point of log n - log p is that if there are n outcomes you can give each a probability of 1/n and that scores zero. You hope to do better and make a positive score, but you can do worse (if something you think really unlikely actually happens)))
The clever mathematics behind this is Gibbs' inequality. When you come up with your probability distribution q, your expect score (relevant if the game has many rounds) is where p is the true but unknown distribution. Gibbs' inequality tells you that this will always be less than
. In the short run you can get a higher score in a particular play of the game by combining overconfidence and good luck :-)
In the long run, you best hope for a high score is to do a good job of guessing p. Scoring log q sets up the game that way.
From Ezra Klein:
Okay, technically, winning the money would be very weak Bayesian evidence that the initial probability estimate was wrong. Still a very good quote.