Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Matt_Simpson comments on The Logic of the Hypothesis Test: A Steel Man - Less Wrong Discussion

5 Post author: Matt_Simpson 21 February 2013 06:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (37)

You are viewing a single comment's thread. Show more comments above.

Comment author: Manfred 21 February 2013 06:53:48AM *  1 point [-]

I hope you informed them about the fact that real people often don't satisfy condition (1). Instead they write things like "The patients in the treatment group showed dramatically decreased risk of stroke (p<0.01), indicating the efficacy of Progenitorivox."

it does seem to be a useful steel man.

Find a steel man in there somewhere, and then we can talk about the properties of said steel man.

Comment author: Matt_Simpson 21 February 2013 06:56:44AM 0 points [-]

The steel man is hypothesis testing as a theory-of-induction free way of doing induction.

Comment author: gjm 21 February 2013 08:42:22AM 10 points [-]

But it isn't theory-of-induction-free. It just pretends to be. There's a theory of induction right in there, where you correctly identify it, in step 4->5. It's no better, no more likely to be true, and no more robust, merely on account of being squashed up small and hidden.

You haven't truly minimized the "amount of induction" in the argument; only the "amount of induction that's easily visible", which I don't think is a parameter that deserves minimizing. You'd need just the same amount of induction if, say, instead of doing classical NHST you did Bayesian inference or maximum likelihood (= Bayesian inference where you pretend not to have priors) or something. You could squash it up just as small, too; you'd just need to make steps 1-4 more quantitative.

Consider a case -- they're not hard to find -- where you have a test statistic that's low-probability on any of your model hypotheses. Then the same logic as you've used says that you're "forced" to conclude that all your hypotheses are false -- even if it happens that one of them is right. (In practice your model is never exactly right, but never mind that.) To me, this shows that the whole enterprise is fundamentally non-deductive, and that trying to make it look as much as possible like a pure deduction is actively harmful.

Comment author: Matt_Simpson 21 February 2013 05:55:29PM 0 points [-]

Maybe a better way of phrasing what I'm trying to point out is that induction is isolated to a single step. Instead of working directly with probabilities which require a theory of what probabilities are, NHST waves it's hands a bit and treats the inductive step as deductive, but transparently so (once you lay out the deduction anyway).

Your point about a test statistic that's low-probability on all possible model hypotheses is a good one - and it suggests that the details of hypothesis testing should change even if the general logic is kept. I doubt that the details of actually used hypothesis testing are ideal for "induction-free induction" (which I'm realizing is a bad name for what I'm trying to convey), but what I'm really talking about is the general logic. I'd be surprised if some of the details didn't have to change.

I don't think I disagree with anything in your comment though. I don't think I have a strong argument for using hypothesis testing, but it may be that the general logic can be salvaged for a reasonable method of doing induction without fully fleshing out an inductive theory (this is why I said one step requires hand waving).