buybuydandavis comments on The Logic of the Hypothesis Test: A Steel Man - Less Wrong

5 Post author: Matt_Simpson 21 February 2013 06:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (37)
Tags: testing induction hypothesis statistics frequentist

Comments (37)

You are viewing a single comment's thread. Show more comments above.

Comment author: buybuydandavis 25 February 2013 04:52:18AM 0 points [-]

The details are hazy at this point, but by assigning a realistic probability to the "Something else" hypothesis, you avoid making over confident estimates of your other hypotheses in a multiple hypothesis testing scenario.

See Multiple Hypothesis Testing in Jaynes PTTLOS, starting pg. 98, and the punchline on pg. 105:

In summary, the role of our new hypothesis C was only to be held in abeyace until needed, like a fire extinguisher. In a normal testing situation, it is "dead", playing no part in the inference because its probability remains far below that of the other hypotheses. But a dead hypothesis can be brought back to life by very unexpected data.

I think this is especially relevant to standard "null hypothesis" hypothesis testing because the likelihood of the data under the alternative hypothesis is never calculated, so you don't even get a hint that your model might just suck, and instead conclude that the null hypothesis should be rejected.

Comment author: jsteinhardt 25 February 2013 06:05:04AM *  0 points [-]

What is the likelihood of the "something else" hypothesis? I don't think this is really a general remedy.

Also, you can get the same thing in the hypothesis testing framework by doing two hypothesis tests, one of which is a comparison to the "something else" hypothesis and one of which is a comparison to the original null hypothesis.

Finally, while I forgot to mention this above, in most cases where hypothesis testing is applied, you actually are considering all possibilities, because you are doing something like P0 = "X <= 0", P1 = "X > 0" and these really are logically the only possibilities =) [although I guess often you need to make some assumptions on the probabilistic dependencies among your samples to get good bounds].

Comment author: buybuydandavis 26 February 2013 12:07:43AM 0 points [-]

Yes, you can say it in that framework. And you should. That's part of the steelmanning exercise - putting in the things that are missing. If you steelman enough, you get to be a good bayesian.

P0 = "X <= 0" and {All My other assumptions}
NOT(P0) = NOT("X <= 0") or NOT({All My other assumptions})

Powered by Reddit Powered by Reddit