You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Dagon comments on Open thread, Mar. 14 - Mar. 20, 2016 - Less Wrong Discussion

3 Post author: MrMind 14 March 2016 08:02AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (212)
Tags: open_thread

Comments (212)

You are viewing a single comment's thread. Show more comments above.

Comment author: Dagon 14 March 2016 03:55:10PM 0 points [-]

The vast majority of yes/no questions you're likely to face won't support 5% intervals. You're just not going to get enough data to have any idea whether the "true" calibration is what actually happens for that small selection of questions.

That said, I agree there's an analytic flaw if you can change true to false on no additional data (kind of: you noticed salience of something you'd previously ignored, which may count as evidence depending on how you arrived at your prior) and only reduce confidence a tiny amount.

One suggestion that may help: don't separate your answer from your confidence confidence, just calculate a probability. Not "true, 60% confidence" (implying 40% unknown, I think, not 40% false), but "80% likely to be true". It really makes updates easier to calculate and understand.

Comment author: ChristianKl 14 March 2016 07:22:38PM 2 points [-]

The vast majority of yes/no questions you're likely to face won't support 5% intervals. You're just not going to get enough data to have any idea whether the "true" calibration is what actually happens for that small selection of questions.

Tetlock found in the Good Judgement Project as described in his book Superforcasting that people who are excellent at forcasting do very finely grained predictions.

Comment author: Luke_A_Somers 14 March 2016 06:59:20PM 1 point [-]

I disagree that you can't get 5% intervals on random yes/no questions - if you stick with 10%, you really only have 5 possible values - 50-59%, 60-69%, 70-79%, 80-89%, and 90+%. That's very coarse-grained.

Comment author: skeptical_lurker 14 March 2016 05:39:05PM *  0 points [-]

The vast majority of yes/no questions you're likely to face won't support 5% intervals.

I agree [edit: actually, it depends on where these yes/no questions are coming from] , but think the questions I was looking at were in the small minority that do support 5% intervals.

Not "true, 60% confidence" (implying 40% unknown, I think, not 40% false)

Perhaps I should have provided more details to explain exactly what I did, because I actually did mean 60% true 40% false.

So, I already was thinking in the manner you advocate, but thanks for the advice anyway!

Powered by Reddit Powered by Reddit