Lumifer comments on Don't You Care If It Works? - Part 1 - Less Wrong

4 Post author: Jacobian 29 July 2015 02:32PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (58)

Comments (58)

You are viewing a single comment's thread. Show more comments above.

Comment author: Lumifer 29 July 2015 07:06:44PM *  2 points [-]

A Bayesian scores himself on total calibration, "number of times my 95% confidence interval includes the truth" is just a small part of it. You can generate an experiment that has a high chance (let's say 99%) of making a Bayesian have a 20:1 likelihood ratio in favor of some hypothesis. By conservation of expected evidence, the same experiment might have 1% chance of generating close to a 2000:1 likelihood ratio against that same hypothesis. A frequentist could never be as sure of anything, this occasional 2000:1 confidence is the Bayesian's reward.

Hold on. Let's say I hire a Bayesian statistician to produce some estimate for me. I do not care about "scoring" or "reward", all I care about is my estimate and how accurate it is. Now you are going to tell me that in 99% of the cases your estimate will be wrong and that's fine because there is a slight chance that you'll be really really sure of the opposite conclusion?

I'm running a Bayesian Casino in Vegas. Debrah Mayo comes to my casino every day with $31.

Why, that's such a frequentist approach X-/

Let's change the situation slightly. You are running the Bayesian Casino and Debrah Mayo comes to you casino once with, say, $1023 in her pocket. Will I lend you money to bet against her? No, I will not. The distribution matters beyond simple expected means.

Comment author: EHeller 29 July 2015 08:50:46PM *  1 point [-]

Reminds of this bit from a Wasserman paper http://ba.stat.cmu.edu/journal/2006/vol01/issue03/wasserman.pdf

van Nostrand: Of course. I remember each problem quite clearly. And I recall that on each occasion I was quite thorough. I interrogated you in detail, determined your model and prior and produced a coherent 95 percent interval for the quantity of interest.

Pennypacker: Yes indeed. We did this many times and I paid you quite handsomely.

van Nostrand: Well earned money I’d say. And it helped win you that Nobel.

Pennypacker: Well they retracted the Nobel and they took away my retirement savings.

... van Nostrand: Whatever are you talking about?

Pennypacker: You see, physics has really advanced. All those quantities I estimated have now been measured to great precision. Of those thousands of 95 percent intervals, only 3 percent contained the true values! They concluded I was a fraud.

van Nostrand: Pennypacker you fool. I never said those intervals would contain the truth 95 percent of the time. I guaranteed coherence not coverage!

Comment author: ike 29 July 2015 09:41:27PM 0 points [-]

Now you are going to tell me that in 99% of the cases your estimate will be wrong

No. Your calibration is still perfect if your priors are perfect. You can only get to that "99% chance of getting strong evidence for hypothesis" if you're already very sure of that hypothesis math here

Powered by Reddit Powered by Reddit