Cyan comments on Case study: abuse of frequentist statistics - Less Wrong

25 Post author: Cyan 21 February 2010 06:35AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (96)

You are viewing a single comment's thread. Show more comments above.

Comment author: cupholder 22 February 2010 03:49:59AM 12 points [-]

Check out the title: abuse of frequentist statistics. Yes, at the end, I argue from a Bayesian perspective, but you don't have to be a Bayesian to see the structural problems with frequentist statistics as currently taught to and practiced by working scientists.

I agree that frequentist statistics are often poorly taught and understood, and that this holds however you like to do your statistics. Still, the main post feels to me like a sales pitch for Bayes brand chainsaws that's trying to scare me off Neyman-Pearson chainsaws by pointing out how often people using Neyman-Pearson chainsaws accidentally cut off a limb with them. (I am aware that I may be the only reader who feels this way about the post.)

(Does the fact that when I saw the sample size the word "underpowered" instantly jumped into my head count as evidence that I am competent?)

Yes, but it is not sufficient evidence to reject the null hypothesis of incompetence at the 0.05 significance level. (I keed, I keed.)

Comment author: Cyan 22 February 2010 04:17:04AM *  2 points [-]

Still, the main post feels to me like a sales pitch...

It's a fair point; I'm not exactly attacking the strongest representative of frequentist statistical practice. My only defense is that this actually happened, so it makes a good case study.

Comment author: cupholder 22 February 2010 04:40:54AM 2 points [-]

That's true, and having been reminded of that, I think I may have been unduly pedantic about a fine detail at the expense of the main point.

Comment author: PhilGoetz 25 February 2010 02:25:43PM 0 points [-]

It's a good case study, but it's not evidence of a problem with frequentist statistics.

Comment author: Cyan 25 February 2010 02:36:05PM *  0 points [-]

I assert that it is evidence in my concluding paragraph, but it's true that I don't give an actual argument. Whether one counts it as evidence would seem to depend on the causal assumptions one makes about the teaching and practice of statistics.

Comment author: PhilGoetz 25 February 2010 10:25:58PM 1 point [-]

Perhaps it's frequentist evidence against frequentist statistics.

Comment author: Cyan 26 February 2010 12:30:07AM 1 point [-]

I think this is just a glib rejoinder, but if there's a deeper thought there, I'd be interested to hear it.

Comment author: PhilGoetz 27 February 2010 04:04:02AM *  2 points [-]

The critique of frequentist statistics, as I understand it - and I don't think I do - is that frequentists like to count things, and trust that having large sample sizes will take care of biases for them. Therefore, a case in which frequentist statistics co-occurs with bad results counts against use of frequentist statistics, and you don't have to worry about why the results were bad.

The whole Bayesian vs. frequentist argument seems a little silly to me. It's like arguing that screws are better than nails. It's true that, for any particular individual joint you wish to connect, a screw will probably connect it more securely and reversibly than a nail. That doesn't mean there's no use for nails.