Cyan comments on Case study: abuse of frequentist statistics - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (96)
I have three answers for that. The first is I wouldn't have bothered -- in this field, data collection is not super-precise, so three data points per method isn't going to tell me shit. But this really avoids the question, so...
The second answer is that Bayesian analogues to this approach do exist. But we know a priori that the two methods won't generate data from the same distribution -- there's really no need to even formulate the null hypothesis. What we really care about is the accuracy and precision of the new method, so...
The third answer is given enough data, I would have set up a hierarchical Bayesian model to estimate the accuracy and precision of the new method, where accuracy is defined as "matching the old method as closely as possible".