In a recent article, John Ioannidis describes a very high proportion of medical research as wrong.
Still, Ioannidis anticipated that the community might shrug off his findings: sure, a lot of dubious research makes it into journals, but we researchers and physicians know to ignore it and focus on the good stuff, so what’s the big deal? The other paper headed off that claim. He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes. Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association.
Part of the problem is that surprising results get more interest, and surprising results are more likely to be wrong. (I'm not dead certain of this-- if the baseline beliefs are highly likely to be wrong, surprising beliefs become somewhat less likely to be wrong.) Replication is boring. Failure to replicate a bright shiny surprising belief is boring. A tremendous amount isn't checked, and that's before you start considering that a lot of medical research is funded by companies that want to sell something.
Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.
Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.
Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.
The culture at LW shows a lot of reliance on small inferential psychological studies-- for example that doing a good deed leads to worse behavior later. Please watch out for that.
A smidgen of good news: Failure to Replicate, a website about failures to replicate psychological findings. I think this could be very valuable, and if you agree, please boost the signal by posting it elsewhere.
From Failure to Replicate's author-- A problem with metastudies:
Eventually, someone else comes across this small literature and notices that it contains “mixed findings”, with some studies finding an effect, and others finding no effect. So this special someone–let’s call them the Master of the Gnomes–decides to do a formal meta-analysis. (A meta-analysis is basically just a fancy way of taking a bunch of other people’s studies, throwing them in a blender, and pouring out the resulting soup into a publication of your very own.) Now you can see why the failure to publish null results is going to be problematic: What the Master of the Gnomes doesn’t know about, the Master of the Gnomes can’t publish about. So any resulting meta-analytic estimate of the association between lawn gnomes and subjective well-being is going to be biased in the positive direction. That is, there’s a good chance that the meta-analysis will end up saying lawn gnomes make people very happy,when in reality lawn gnomes only make people a little happy, or don’t make people happy at all.
The people I've read who gave advice based on Ioannidis article strongly recommended eating paleo. I don't think this is awful advice in the sense that a number of people seem to actually feel better following it, and I haven't heard of disasters resulting from eating paleo. However, I don't know that it's a general solution to the problems of living with a medical system which does necessary work some of the time, but also is wildly inaccurate and sometimes destructive.
The following advice is has a pure base of anecdote, but at least I've heard a lot of them from people with ongoing medical problems. (Double meaning intended.)
Before you use prescription drugs and/or medical procedures, make sure there's something wrong with you. Keep an eye out for side effects and the results of combined medicines. Check for evidence that whatever you're thinking about doing actually helps. Be careful with statins-- they can cause reversible memory problems and permanent muscle weakness. Choose a doctor who listens to you.
Forum about self-experimentation-- note: even Seth Roberts is apt to oversell his results as applying to everyone.
Link about the failure to replicate site found here.
Okay, but don't make the mistake of the guy who says "The mainstream media is all lies - so I'll only trust what I read on shady Internet conspiracy sites". Saying that there are likely flaws in mainstream medical research doesn't license you to discount any specific medical finding unless you have particular reason to believe that finding is false. And it certainly doesn't license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories. Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
The study mentioned above looks at exciting cutting-edge research over the past decade. It says that 40% or so was proven wrong. This is good and to the credit of medical science! It means the system is working as it should in retesting things and getting the false stuff out. The basis of science isn't getting everything right the first time, it's making sure everyone's work gets checked and double-checked until only the truth survives. An unreplicated study in almost any area is an intriguing possibility and nothing more; medicine is no exception. If the media makes a big deal about a new study and publishes "VITAMIN B CURES BREAST CANCER!!!" in 72 point font in the newspapers, that is an interesting fact about the media and the people who believe it, but not an interesting fact about medical science.
Good doctors are both conservative and utilitarian. They stick to older, well-proven treatments unless the advantage of a new treatment is so great that it outweighs the uncertainty and risks involved. IMHO the medical consensus has been right on the important things a surprising amount of the time.
I would strongly discourage people from bewaring statins overly much. I don't see anything by Ioannidis saying the studies surrounding statins are particularly bad. Ioannidis says research is less likely to be true if it has low sample sizes, low effect sizes, bias, and a wide net. There have been several statin trials with sample sizes in the thousands to tens of thousands (see: JUPITER, SSSS, etc.) They've found that death rate from heart attacks in people correctly prescribed statin goes down by 30%, which is not at all a small effect size. Many such trials have not been linked to statin manufacturers or anyone with an axe to grind. And because people already know statins are supposed to reduce cholesterol, there is much less of a wide net than if you were to give a bunch of people statins and, say, see if any diseases became less common - the studies had a clearly designated endpoint, which they achieved.
Are there people who suggest the side effects of statins are worse than everyone else thinks? A few, and based off of very little evidence (I believe the idea that statins cause memory dysfunction is based mostly off isolated case reports, and there are only 60 out of many years of hundreds of thousands of people on statins - basically background noise). I haven't investigated this thoroughly, but the side effects would have to be pretty darned bad and pretty darned robust to stop prescribing a drug with an NNT in the two digits (ie it takes under 100 statin prescriptions to prevent one heart attack), and I treat people trying to exaggerate drug side effects as just as real a failure mode as doctors trying to exaggerate drug benefits, and use just as much caution.
The advice in the third-to-last paragraph, except perhaps the specific singling out of statins, remains excellent.
Overall, the use of the term "license" here raises yellow flags for me (see Hero Licensing for the basic reason). It conflates social standing with epistemic standing. The first paragraph here seems a bit confused in other ways too, let me try to break it up into what I see as comparatively crisp distinct claims.
CLAIM: Saying that there are likely flaws in mainstream medical research [...] doesn't license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that... (read more)