= 27d567bc2d48d9f40e9ccc20ea370b15)
G0W51 comments on Open thread, Mar. 9 - Mar. 15, 2015 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (109)
Perhaps it would be beneficial to make a game used for probability calibration in which players are asked questions and give answers along with their probability estimate of it being correct. The number of points gained or lost would be a function of the player’s probability estimate such that players would maximize their score by using an unbiased confidence estimate (i.e. they are wrong p proportion of the time when they say they think they are correct with probability p. I don’t know of such a function off hand, but they are used in machine learning, so they should be able to be found easily enough. This might already exist, but if not, it could be something CFAR could use.
One function that works for this is log scoring: the number of points you get is the log of the probability you place in the correct answer. The general thing to google to find other functions that work for this is "log scoring rules".
At the Australian mega-meetup, we played the standard 2-truths-1-lie icebreaker game, except participants had to give their probability for each statement being the lie, and were given log scores. I can't answer for everybody, but I thought it was quite fun.
It exists as the credence game.
Hey, we can deconstruct Doyle's Sherlock Holmes stories, assigning probabilities to every single inference and offering alternative explanations. Or take some other popular fiction. That might also help people who, like me, struggle with counterfactuals.