army1987 comments on How to Fix Science - Less Wrong

50 Post author: lukeprog 07 March 2012 02:51AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (141)

You are viewing a single comment's thread. Show more comments above.

Comment author: satt 04 March 2012 01:38:35AM *  6 points [-]

Bayesian methods are better in a number of ways, but ignorant people using a better tool won't necessarily get better results. I don't think the net effect of a mass switch to Bayesian methods would be negative, but I do think it'd be very small unless it involved raising the general statistical competence of scientists.

Even when Bayesian methods get so commonplace that they could be used just by pushing a button in SPSS, researchers will still have many tricks at their disposal to skew their conclusions. Not bothering to publish contrary data, only publishing subgroup analyses that show a desired result, ruling out inconvenient data points as "outliers", wilful misinterpretation of past work, failing to correct for doing multiple statistical tests (and this can be an issue with Bayesian t-tests, like those in the Wagenmakers et al. reanalysis lukeprog linked above), and so on.

Comment author: [deleted] 11 March 2012 01:19:29AM *  0 points [-]

I don't think the net effect of a mass switch to Bayesian methods would be negative, but I do think it'd be very small unless it involved raising the general statistical competence of scientists.

I think that teaching Bayesian methods would itself raise the general statistical competence of scientists as a side effect, among other things because the meaning of p-values is seriously counter-intuitive (so more scientists would actually grok Bayesian statistics in such a world than actually grok frequentist statistics right now).

Comment author: satt 11 March 2012 02:36:46PM 1 point [-]

You could well be right. I'm pessimistic about this because I remember seeing lots of people at school & university recoiling from any statistical topic more advanced than calculating means and drawing histograms. If they were being taught about conjugate priors & hyperparameters I'd expect them to react as unenthusiastically as if they were being taught about confidence levels and maximum likelihood. But I don't have any rock solid evidence for that hunch.