= 27d567bc2d48d9f40e9ccc20ea370b15)
Psy-Kosh comments on Case study: abuse of frequentist statistics - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (96)
Yes, but I don't understand from the details presented why that follows. Why couldn't result X contain the new method ranking below the standard method?
What the OP was saying was that this test only depends on the rankings. So to check for sanity, he calculated what the p values would have been for all possible rankings and found that none of those p values would be below .05.
In other words, it was a mathematical impossibility for this test, when treated this way, to result in a rejection of the null hypothesis. There was no possible outcome given this many data points, analyzed using this method, a rejection.
(in other words, it was a "heads I win, tails you lose" situation)
Okay, I think that makes sense. Let me put it into my own words:
The test is guaranteed to be not statistically significant merely by virtue of cutting up the outcome space into pieces, each of which has at least 5% chance of happening. And further, because the null hypothesis has been (arbitrarily) defined to be "the two methods are the same", statistical insignificance means a favorable result.
Does that about cover it? If so, that's pretty bad.
That part isn't right, but the rest is.
So I should have said "for the nine outcomes they considered, they all had at least 5% chance of happening"?
The p-value is the probability of getting a result "at least this extreme" given the null hypothesis, where "extreme" means "deviating from the null hypothesis", however that's defined. So, the test cut the outcome space into pieces, the most extreme of which had at least a 5% chance of happening.
I think.
... under the null hypothesis. I actually forgot this detail when replying to komponisto.
Wait... actually it may even be worse than that. I'm not even sure it's cleanly partitioning the outcome space. 1/20 = .05, so if some outcomes are above .05, then other outcomes would have to be below .05, right?
So the calculation to get the final result doesn't even really do a proper partitioning of the outcomes if some of the outcomes can be greater than .05 and none less than .05
EDIT: so yeah, it's cutting up not just the outcome space into pieces corresponding to rankings, but mushing some of those together (at best).
That's more or less my understanding of the situation.
And yes... that is indeed pretty bad. :)
More of a double-headed coin situation, actually.
Well... different ranking outcomes (different sides of the coin) are possible. Just that the interpretation will always be "don't reject the null hypothesis" but yeah. :)
Either way, my overall reaction to your post is "yuck" (not your post itself! That I upvoted. I mean the whole situation... That a relatively standard statistical test could allow this sort of madness. I mean, I know frequentist stats isn't the Bayesian way, but that relatively standard methods in it can be this pathological does not at all give me warm fuzzies)
I concur with your "yuck", but would phrase it slightly differently. The specific type of statistical test applied, plus the number of samples taken, has the effect, as Cyan said, of guaranteeing the results that the authors wanted. Note that, more generally, the fact that the authors chose to phrase their analysis so that accepting the null hypothesis was the result they wanted plus choosing a nonparametric statistical test, which is always weaker than a parametric one is in and of itself suspicious. If they had had enough samples so that it would be theoretically possible for the null hypothesis to be rejected (say if they had taken more samples) but they had still wanted the null result and they had still chosen a nonparametric test I would still be suspicious. As Cyan said, the nonparametric tests throw away most of the information.
It's not the fault of the method if someone abuses it.
In general, no. However, if a method is more easily abused than others, that that is something worth pointing out.