I like the idea of defining a betting game 'forecasters vs cosmic bookie'. Then saying 'the probability that people will land on Mars by 2040 is 42%' translates into semantics 'I am willing to buy an option for Y<42 cents that would be worth $1 if we land on Mars by 2040 or $0 otherwise'.
To compare several forecasters we can consider a game in which each player is offered to buy some options of this kind. Suppose that for each x in {1, \dots, 99} each player is allowed to buy one option for x cents. If one believes that the probability of an event is 30... (read more)
That's perfect, I was thinking along the same lines, with a range of options available for sale, but didn't do the math and so didn't realize the necessity of dual options. And you are right of course, there's still quite a bit of arbitrariness left. In addition to varying the distribution of options there is, for example, freedom to choose what metric the forecasters are supposed to optimize. It doesn't have to be EV, in fact in real life it rarely should be EV, because that ignores risk aversion. Instead we could optimize some utility function that becomes flatter for larger gains, for example we could use Kelly betting.
I like the idea of defining a betting game 'forecasters vs cosmic bookie'. Then saying 'the probability that people will land on Mars by 2040 is 42%' translates into semantics 'I am willing to buy an option for Y<42 cents that would be worth $1 if we land on Mars by 2040 or $0 otherwise'.
To compare several forecasters we can consider a game in which each player is offered to buy some options of this kind. Suppose that for each x in {1, \dots, 99} each player is allowed to buy one option for x cents. If one believes that the probability of an event is 30... (read more)