Matt_Simpson comments on The Logic of the Hypothesis Test: A Steel Man - Less Wrong

5 Post author: Matt_Simpson 21 February 2013 06:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (37)

You are viewing a single comment's thread. Show more comments above.

Comment author: Matt_Simpson 26 February 2013 08:38:56PM *  0 points [-]

Thanks for this comment. I was attempting to abstract away from the specific details of NHST and talk about the general idea since in many particulars there is much to criticize, but it appears that I abstracted too much - the ordering of the hypothesis space (i.e. a monotone likelihood ratio as in Neyman-Pearson) is definitely necessary.

In a closely analogous fashion, a hypothesis test is a probe for a certain kind of inadequacy in the statistical model. The statistic is the equivalent of the grade, and the threshold of statistical significance is equivalent of the standard of bare adequacy.

This seems to back up my claim that we can still view NHST as a sort of induction without a detailed theory of induction (though the reasons for and nature of this "thin" induction must be different from what I was thinking about). Do you agree?

Comment author: Cyan 28 February 2013 12:50:14AM 0 points [-]

I agree that the quote seems to back up the claim, but I don't agree with the claim. Like all frequentist procedures, NHST does have a detailed theory of induction founded on the notion that one can use just the (model's) sampling probability of a realized event to generate well-warranted claims about some hypothesis/hypotheses. (Again, see the work of Deborah Mayo.)