brian_jaress comments on Case study: abuse of frequentist statistics - Less Wrong

25 Post author: Cyan 21 February 2010 06:35AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (96)
Tags: frequentism bayesian statistics madness

Comments (96)

You are viewing a single comment's thread. Show more comments above.

Comment author: Cyan 22 February 2010 03:26:57AM *  1 point [-]

Thanks for the pointer to the original paper.

I'm not seeing why what you call "the real WTF" is evidence of a problem with frequentist statistics.

Check out the title: abuse of frequentist statistics. Yes, at the end, I argue from a Bayesian perspective, but you don't have to be a Bayesian to see the structural problems with frequentist statistics as currently taught to and practiced by working scientists.

I would hope that any competent statistician, frequentist or not, would be sceptical of a nonparametric comparison of means for samples of size 3!

Me too. But not all papers with shoddy statistics are sent to statisticians for review. Experimental biologists in particular have a reputation for math-phobia. (Does the fact that when I saw the sample size the word "underpowered" instantly jumped into my head count as evidence that I am competent?)

Comment author: brian_jaress 23 February 2010 06:02:49PM 3 points [-]

I think that, in this case, the underlying problem was not caused by the way frequentist statistics are commonly taught and practiced by working scientists:

In the present case, the null hypothesis is that the old method and the new method produce data from the same distribution; the authors would like to see data that do not lead to rejection of the null hypothesis.

I'm no statistician, but I'm pretty sure you're not supposed to make your favored hypothesis the null hypothesis. That's a pretty simple rule and I think it's drilled into students and enforced in peer review.

I see that as the underlying problem because it reverses the burden of proof. If they had done it the right way around, six data points would have been not enough to support their method instead of being not enough to reject it. Making your favored hypothesis the null hypothesis can allow you, in the extreme, to rely on a single data point.

Comment author: Cyan 23 February 2010 06:18:08PM *  1 point [-]

In the OP I did refer to that when I wrote:

Now even from a frequentist perspective, this is wacky. Hypothesis testing can reject a null hypothesis, but cannot confirm it, as discussed in the first paragraph of the Wikipedia article on null hypotheses.

You wrote:

That's a pretty simple rule and I think it's drilled into students and enforced in peer review.

Not all papers are reviewed by people who know the rule. I was taught that rule over ten years ago, and I didn't remember it when my colleague showed me the analysis. (I did recall it eventually, just after I ran the sanity check. Evidence against my competence!) My colleague whose job it was to review the paper didn't know/recall the rule either.

Powered by Reddit Powered by Reddit