Will_Newsome comments on How to Fix Science - Less Wrong

50 Post author: lukeprog 07 March 2012 02:51AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (141)

You are viewing a single comment's thread. Show more comments above.

Comment author: satt 04 March 2012 01:38:35AM *  6 points [-]

Bayesian methods are better in a number of ways, but ignorant people using a better tool won't necessarily get better results. I don't think the net effect of a mass switch to Bayesian methods would be negative, but I do think it'd be very small unless it involved raising the general statistical competence of scientists.

Even when Bayesian methods get so commonplace that they could be used just by pushing a button in SPSS, researchers will still have many tricks at their disposal to skew their conclusions. Not bothering to publish contrary data, only publishing subgroup analyses that show a desired result, ruling out inconvenient data points as "outliers", wilful misinterpretation of past work, failing to correct for doing multiple statistical tests (and this can be an issue with Bayesian t-tests, like those in the Wagenmakers et al. reanalysis lukeprog linked above), and so on.

Comment author: Will_Newsome 04 March 2012 03:07:17AM 1 point [-]

ISTM a large benefit of commonplace Bayes would be that competent statisticians could do actually meaningful meta-analyses...? Which would counteract widespread statistical ineptitude to a significant extent...?

Comment author: satt 04 March 2012 11:45:27AM *  2 points [-]

I'm not sure it'd make much difference. From reading & skimming meta-analyses myself I've inferred that the main speedbumps with doing them are problems with raw data themselves or a lack of access to raw data. Whether the data were originally summarized using NHST/frequentist methods or Bayesian methods makes a lot less difference.

Edit to add: when I say "problems with raw data themselves" I don't necessarily mean erroneous data; a problem can be as mundane as the sample/dataset not meeting the meta-analyst's requirements (e.g. if the sample were unrepresentative, or the dataset didn't contain a set of additional moderator variables).