= 27d567bc2d48d9f40e9ccc20ea370b15)
AlexSchell comments on Open Thread, November 1-15, 2012 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (373)
Possible idea for a post:
There isn't much material here on the problem of multiple comparisons. This is something that humans routinely stumble over, while for an ideal Bayesian it wouldn't even be a problem requiring a solution (much like e.g. confirmation bias). The post would describe the multiple comparisons problem, explain why it's a non-issue for Bayesians, and look into plausible candidates for the psychological mechanisms that give rise to it (hindsight bias, privileging the hypothesis, base-rate neglect; any others?).
Reply here if you are (actually) starting to work on this.
I'd love to see a post on this, ideally with R code. In particular, I need to know about this because I'm running a big sleep experiment with 5 separate interventions, each with multiple endpoints. You can see the problem.
I've done multiple correction of p-values with my previous frequentist analyses with the same problem of multiple endpoints, but I'd rather do a Bayesian analysis; however, I don't know how to do multiple correction with Bayesian results. Reading, a Gelman paper tells me that I don't need to because if I'm doing hierarchical models, probability mass gets automatically reallocated across models and obviates the need for correction - whatever that means, not that I know hierarchical models either!
My idea was less about statistical practice than about very simple toy models illustrating some general points (in particular, if you write down your priors beforehand and use likelihood ratios, you can do as many comparisons as you like, without any 'adjustments'; the reason multiple comparisons are suspect in practice has to do with human biases and the circumstances under which scientists will engage in this sort of data mining). I've since read a paper that makes pretty much the same theoretical points, although it overstates their practical significance.