gwern comments on Alternative to Bayesian Score - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (29)
Hm, no, I wasn't really thinking that way. I don't want some finite number, I want everyone to reach different numbers so more accurate predictors score higher.
The weights on particular functions do not have to be even algorithmicly set - for example, a prediction market is immune to the 'sky is blue' problem because if one were to start a contract for 'the sky is blue tomorrow', no one would trade on it unless one were willing to lose money being a market-maker as the other trader bid it up to the meteorologically-accurate 80% or whatever. One can pick and choose as much as one pleases, but unless one's contracts were valuable to other people for any reason, it would be impossible to make money by stuffing the market with bogus contracts. The utility just becomes how much money you made.
I think this doesn't work because you're trying to invent a non-informative prior, and it's trivial to set up sets of predictions where the obviously better non-informative prior is not 1/2: for example, set up 3 predictions for each of 3 mutually-exhaustive outcomes, where the non-informative prior obviously looks more like 1/3 and 1/2 means someone is getting robbed. More importantly, uninformative priors are disputed and it's not clear what they are in more complex situations. (Frequentist Larry Wasserman goes so far as to call them "lost causes" and "perpetual motion machines".)
Perhaps a raw log odds is not the best idea, but do you really think there is no way to interpret them into some score which disincentivizes strategic predicting? This sounds just arrogant to me, and I would only believe it if you summarized all the existing research into rewarding experts and showed that log odds simply could not be used in any circumstance where any predictor could predict a subset of the specified predictions.
There aren't finitely many questions because one can ask questions involving each of the infinite set of integers... Knowing that questions are asking identical questions sounds like an impossible demand to meet (for example, if any system claimed this, it could solve the Halting Problem by simply asking it to predict the output of 2 Turing machines).