Johnicholas comments on Case study: abuse of frequentist statistics - Less Wrong

25 Post author: Cyan 21 February 2010 06:35AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (96)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 22 February 2010 07:10:19AM *  7 points [-]

I like your point but not your example.

Suppose (because you're a computationally-limited Bayesian) that you only include in your model the N highest-probability hypotheses. That is, you include A, B, C, in your model, but you neglect Z - that is, you put zero probability on it. (We can make Z's pre-evidence probability arbitrarily small, to make this seem reasonable at the time.) When one, or even N balls turn out to be labeled Z, the model (due to the initial zero probability on Z) continues insisting that the balls came from one of the initially-specified hypotheses.

That isn't just a computational limitation. It's an outright bug. Something that assigns 0 to Z is just not even an approximation of a Bayesian. A sane agent with limited resources may, for example, assign a probability to "A,B,C and 'something else'". If it explicitly assigned an (arbitrarily close to) 0 to Z then it just fails at life.

Comment author: Johnicholas 28 February 2010 12:24:28AM 3 points [-]

Hi. I found the paper containing the example in question - it's Bayesians sometimes cannot ignore even very implausible theories. I don't understand everything in the paper, but it seems like they've anticipated your objection and have another example which explicitly includes a "Something else" case.