curiousepic comments on Why epidemiology will not correct itself - Less Wrong

38 Post author: gwern 11 August 2011 12:54AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (13)

You are viewing a single comment's thread. Show more comments above.

Comment author: Andy_McKenzie 11 August 2011 03:44:09AM 10 points [-]

This is too nihilistic and is not really what experts like Ioannidis are proposing. Better to evaluate the studies (or find sources that evaluate the studies) individually for their sample size and statistical measures, such as whether or not they control for relevant covariates and do multiple hypothesis testing corrections.

You can download a video of Ioannidis' Mar '11 lecture on nutrition from http://videocast.nih.gov/PastEvents.asp?c=144 (it's big though, 250 MB). Some notes:

  • Randomized trials have problems too.
  • For example, they'll often inflate the effects by contrasting the most extreme groups (upper vs lower 20%).
  • Or, just basic biases, like the winner's curse (large effects tend to come from studies with small sample sizes--you can see this by comparing the log of treatment effect vs the log of total sample size in the cochrane database) or publication bias (leading to missing data).
  • Odds ratios in randomized trials also decrease over time.

Generally, Ioannidis wants massive testing via biobanks (sample sizes in the millions), longitudinal measurements, and large-scale global collaborations. These do not necessarily mean only randomized trials, and in fact they are pretty much impossible for that kind of data set. Epi can work too, it just needs to be done well.

Comment author: curiousepic 11 August 2011 02:24:55PM 3 points [-]

Recently it was announced that some organization (It thought it was the SIAI but i can't find it in their blog) would work to form a panel in order to examine and disambiguate the state of knowledge about a number of different areas, the first being diet, nutrition and exercise. It seems imperative that they take this into consideration. What was this organization, and do we have any way of knowing whether they will or not?

Comment author: XFrequentist 11 August 2011 05:32:43PM 6 points [-]

Are you referring to the Persistent Problems Group?

Comment author: gwern 11 August 2011 06:30:02PM 2 points [-]

My own opinion of that proposal (I'm not sure whether I said this elsewhere) is that the Group is already being done, and better, by things like the Cochrane Collaboration. There is no comparative advantage there.

Comment author: XFrequentist 11 August 2011 07:48:40PM *  1 point [-]

That was my thought as well, although if this group were formed I'd be extremely interested in how they worked and what their findings were. I'd imagine Bayesian methods would be the norm, which might give them a leg up.

It would be particularly interesting if they consistently disagreed with mainstream systematic reviews.

Comment author: curiousepic 11 August 2011 11:48:15PM 0 points [-]

Yes, thanks.