soreff comments on Case study: abuse of frequentist statistics - Less Wrong

25 Post author: Cyan 21 February 2010 06:35AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (96)

You are viewing a single comment's thread. Show more comments above.

Comment author: soreff 23 February 2010 10:49:02PM *  1 point [-]

I concur with your "yuck", but would phrase it slightly differently. The specific type of statistical test applied, plus the number of samples taken, has the effect, as Cyan said, of guaranteeing the results that the authors wanted. Note that, more generally, the fact that the authors chose to phrase their analysis so that accepting the null hypothesis was the result they wanted plus choosing a nonparametric statistical test, which is always weaker than a parametric one is in and of itself suspicious. If they had had enough samples so that it would be theoretically possible for the null hypothesis to be rejected (say if they had taken more samples) but they had still wanted the null result and they had still chosen a nonparametric test I would still be suspicious. As Cyan said, the nonparametric tests throw away most of the information.