Is there a way to incentivize reporting true probability distribution? Say I Bob wants Alice to provide her probability distribution of IQ she'll get on the test. He is willing to give her a real number as a reward, he wants to hear her probability distribution of her result. What should he do?
Would be nice if it worked for both discrete and non-discrete probability spaces.
In the discrete case log scoring still works, it generalizes past the binary case.
That is, if S is the set of possible outcomes of the test, Bob elicits from Alice a probability distribution q(s) on S, then Alice takes the test and gets some outcome s∈S, then Bob rewards Alice logq(s). (This number is unfortunately always negative; you can add a positive constant to it if you want.)
Alice's expected payoff according to her true probability distribution p(s) is
∑s∈Sp(s)logq(s)
also known as the (negative of the) cross entropy between p and q. And you can do a computation, e.g. with Lagrange multipliers, which will verify that for fixed p, the optimal value of q is q=p. I do this calculation in this blog post.
A test isn't a good example to use because the outcome of the test is under Alice's control, so she can e.g. throw the test and predict this fact. This procedure is best used to elicit Alice's prediction of something which she cannot influence in any way.
Is there a way to incentivize reporting true probability distribution? Say I Bob wants Alice to provide her probability distribution of IQ she'll get on the test. He is willing to give her a real number as a reward, he wants to hear her probability distribution of her result. What should he do?
Would be nice if it worked for both discrete and non-discrete probability spaces.