ike comments on Don't You Care If It Works? - Part 1 - Less Wrong

4 Post author: Jacobian 29 July 2015 02:32PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (58)

You are viewing a single comment's thread. Show more comments above.

Comment author: Lumifer 29 July 2015 07:06:44PM *  2 points [-]

A Bayesian scores himself on total calibration, "number of times my 95% confidence interval includes the truth" is just a small part of it. You can generate an experiment that has a high chance (let's say 99%) of making a Bayesian have a 20:1 likelihood ratio in favor of some hypothesis. By conservation of expected evidence, the same experiment might have 1% chance of generating close to a 2000:1 likelihood ratio against that same hypothesis. A frequentist could never be as sure of anything, this occasional 2000:1 confidence is the Bayesian's reward.

Hold on. Let's say I hire a Bayesian statistician to produce some estimate for me. I do not care about "scoring" or "reward", all I care about is my estimate and how accurate it is. Now you are going to tell me that in 99% of the cases your estimate will be wrong and that's fine because there is a slight chance that you'll be really really sure of the opposite conclusion?

I'm running a Bayesian Casino in Vegas. Debrah Mayo comes to my casino every day with $31.

Why, that's such a frequentist approach X-/

Let's change the situation slightly. You are running the Bayesian Casino and Debrah Mayo comes to you casino once with, say, $1023 in her pocket. Will I lend you money to bet against her? No, I will not. The distribution matters beyond simple expected means.

Comment author: ike 29 July 2015 09:41:27PM 0 points [-]

Now you are going to tell me that in 99% of the cases your estimate will be wrong

No. Your calibration is still perfect if your priors are perfect. You can only get to that "99% chance of getting strong evidence for hypothesis" if you're already very sure of that hypothesis math here