You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

jsteinhardt comments on The Logic of the Hypothesis Test: A Steel Man - Less Wrong Discussion

5 Post author: Matt_Simpson 21 February 2013 06:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (37)
Tags: testing induction hypothesis statistics frequentist

Comments (37)

You are viewing a single comment's thread. Show more comments above.

Comment author: jsteinhardt 25 February 2013 03:11:53AM 1 point [-]

Nope. This was a good point by Jaynes. The truth may not exist in your hypothesis space. It may be (and often is) something you haven't conceived of.

If the truth doesn't exist in your hypothesis space then Bayesian methods are just as screwed as frequentist methods. In fact, Bayesian methods can grow increasingly confident that an incorrect hypothesis is true in this case. I don't see how this is a weakness of Matt's argument.

Comment author: buybuydandavis 25 February 2013 04:52:18AM 0 points [-]

The details are hazy at this point, but by assigning a realistic probability to the "Something else" hypothesis, you avoid making over confident estimates of your other hypotheses in a multiple hypothesis testing scenario.

See Multiple Hypothesis Testing in Jaynes PTTLOS, starting pg. 98, and the punchline on pg. 105:

In summary, the role of our new hypothesis C was only to be held in abeyace until needed, like a fire extinguisher. In a normal testing situation, it is "dead", playing no part in the inference because its probability remains far below that of the other hypotheses. But a dead hypothesis can be brought back to life by very unexpected data.

I think this is especially relevant to standard "null hypothesis" hypothesis testing because the likelihood of the data under the alternative hypothesis is never calculated, so you don't even get a hint that your model might just suck, and instead conclude that the null hypothesis should be rejected.

Comment author: jsteinhardt 25 February 2013 06:05:04AM *  0 points [-]

What is the likelihood of the "something else" hypothesis? I don't think this is really a general remedy.

Also, you can get the same thing in the hypothesis testing framework by doing two hypothesis tests, one of which is a comparison to the "something else" hypothesis and one of which is a comparison to the original null hypothesis.

Finally, while I forgot to mention this above, in most cases where hypothesis testing is applied, you actually are considering all possibilities, because you are doing something like P0 = "X <= 0", P1 = "X > 0" and these really are logically the only possibilities =) [although I guess often you need to make some assumptions on the probabilistic dependencies among your samples to get good bounds].

Comment author: buybuydandavis 26 February 2013 12:07:43AM 0 points [-]

Yes, you can say it in that framework. And you should. That's part of the steelmanning exercise - putting in the things that are missing. If you steelman enough, you get to be a good bayesian.

P0 = "X <= 0" and {All My other assumptions}
NOT(P0) = NOT("X <= 0") or NOT({All My other assumptions})

Powered by Reddit Powered by Reddit