Okay, but don't make the mistake of the guy who says "The mainstream media is all lies - so I'll only trust what I read on shady Internet conspiracy sites". Saying that there are likely flaws in mainstream medical research doesn't license you to discount any specific medical finding unless you have particular reason to believe that finding is false. And it certainly doesn't license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories. Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
The study mentioned above looks at exciting cutting-edge research over the past decade. It says that 40% or so was proven wrong. This is good and to the credit of medical science! It means the system is working as it should in retesting things and getting the false stuff out. The basis of science isn't getting everything right the first time, it's making sure everyone's work gets checked and double-checked until only the truth survives. An unreplicated study in almost any area is an intriguing possibility and nothing more; medicine is no exception. If the media makes a big deal about a new study and publishes "VITAMIN B CURES BREAST CANCER!!!" in 72 point font in the newspapers, that is an interesting fact about the media and the people who believe it, but not an interesting fact about medical science.
Good doctors are both conservative and utilitarian. They stick to older, well-proven treatments unless the advantage of a new treatment is so great that it outweighs the uncertainty and risks involved. IMHO the medical consensus has been right on the important things a surprising amount of the time.
I would strongly discourage people from bewaring statins overly much. I don't see anything by Ioannidis saying the studies surrounding statins are particularly bad. Ioannidis says research is less likely to be true if it has low sample sizes, low effect sizes, bias, and a wide net. There have been several statin trials with sample sizes in the thousands to tens of thousands (see: JUPITER, SSSS, etc.) They've found that death rate from heart attacks in people correctly prescribed statin goes down by 30%, which is not at all a small effect size. Many such trials have not been linked to statin manufacturers or anyone with an axe to grind. And because people already know statins are supposed to reduce cholesterol, there is much less of a wide net than if you were to give a bunch of people statins and, say, see if any diseases became less common - the studies had a clearly designated endpoint, which they achieved.
Are there people who suggest the side effects of statins are worse than everyone else thinks? A few, and based off of very little evidence (I believe the idea that statins cause memory dysfunction is based mostly off isolated case reports, and there are only 60 out of many years of hundreds of thousands of people on statins - basically background noise). I haven't investigated this thoroughly, but the side effects would have to be pretty darned bad and pretty darned robust to stop prescribing a drug with an NNT in the two digits (ie it takes under 100 statin prescriptions to prevent one heart attack), and I treat people trying to exaggerate drug side effects as just as real a failure mode as doctors trying to exaggerate drug benefits, and use just as much caution.
The advice in the third-to-last paragraph, except perhaps the specific singling out of statins, remains excellent.
Overall, the use of the term "license" here raises yellow flags for me (see Hero Licensing for the basic reason). It conflates social standing with epistemic standing. The first paragraph here seems a bit confused in other ways too, let me try to break it up into what I see as comparatively crisp distinct claims.
CLAIM: Saying that there are likely flaws in mainstream medical research [...] doesn't license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories.
This seems basically true, and is an application of Beware Isolated Demands for Rigor.
CLAIM: Saying that there are likely flaws in mainstream medical research doesn't license you to discount any specific medical finding unless you have particular reason to believe that finding is false.
The burden of evidence is on the people making claims that constrain our anticipations. We can and should discount all statistical findings in proportion to the evidence that they're unreliable.
CLAIM: Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
Statistics isn't the only sort of evidence. "The same high standards" isn't necessarily meaningful here, as different kinds of standards are required for different kinds of evidence. There's also checking a claim against your model of how the world works, and trying things to see whether they produce the claimed effect. The Paleo argument matches my underlying understanding of how the world works, while I'd find the opposite claim pretty counterintuitive. Then I tried eating vaguely Paleo and I felt noticeably better and lost 20 pounds. I have a bunch of friends who report vaguely similar results. Anecdotes like this don't overrule strong statistical evidence, but that doesn't mean much when there's not strong statistical evidence. A little evidence is often better than none, and "standards" may not be the right paradigm here.
It's nice to be careful not to overgeneralize from personal experience, and it's also nice to be careful not to overgeneralize from unreplicated underpowered studies, especially when they find something counterintuitive that contradicts my life experience or would be hard for me to check.
That's a very good point -- medical science being bad does not imply that your favorite contrarian idea is good.
So if I understand you right, the moral that we should draw is that the most reliable treatments are the old ones. A new study constitutes very weak evidence in favor of anything.
I remain skeptical of medicine, but there is a particular kind of emotional/intellectual scam that is practiced by some alternative medicine practitioners, akin to what Yvain cautions.
Particularly when the standard, tested regimen (say chemo + radiation for cancer) has miserable side effects, it's tempting to go instead for the herbal remedy (or the like).
But until those alternative treatments are subjected to large trials, what we're trading, basically, is the flawed, but at least somewhat familiar and certainly broad-based approach of medical research for the much murkier world of individual "authorities" and anecdotes. It's darkly ironic that the same folks who urge me to discount the authority of the medical establishment are eager to have me listen to their "authorities" instead.
The emotional (and generally unacknowledged) part of this scam is that the non-traditional recipe is particularly appealing when the standard remedy is horrible.
You may be right about the statins-- part of what spooked me about them was running into a woman whose husband had taken permanent muscle damage from them, which suggested to me that the side effect might not be all that rare.
You mentioned that it was important for them to be correctly prescribed. How common is it for them to not be correctly prescribed?
Rhabdomyolysis, which I think is the kind of severe permanent muscle damage you're talking about, is well-known enough as a side effect of statins that it's taught in first year medical school classes. There was one statin that may have had a relatively high (1/2,000 per year) rhabdomyolysis rate and was withdrawn from the market after a couple of years for that reason. The statins currently on the market have about a 1/20,000/year rhabdomyolysis rate, which is actually low enough that no one is entirely sure it's not background noise although no one's taking any chances. Since they also have a 1+/500/year heart attack prevention rate, they prevent something like 50 heart attacks for each case of rhabdomyolysis they cause, which seems "worth it".
Muscle damage rates increase by a lot if you take statins with fibrates (another cholesterol lowering drug). I think (not sure) that prescribing these two drugs together is Officially Discouraged, although there might be some leeway in cases of people with crazy high cholesterol. I've also heard having grapefruit juice with statins increases the risk (grapefruit juice messes with liver enzymes) but I'm not sure if that is practically important or just random clinical trivia.
As for correct prescription: I am only a student, I haven't checked the official guidelines, and if you hear otherwise from any doctor trust the doctor and not me - however, as I understand it there is pretty good evidence for giving a statin to people who have already had a cardiovascular event in order to prevent a second one, and much weaker evidence (depending on whose studies and meta-analyses you prefer) for giving it to someone who's never had a cardiovascular event. Many doctors give them to the latter category anyway just because irreversible side effects are so rare and they would rather be safe than sorry regarding heart attacks; I see some merit in both sides of the argument.
I've had a little more time to think-- how sure are you that the studies you cite were well-constructed?
My only evidence is that they're mostly by large and respected institutions, considered exemplary by the medical community, and that reading a one page summary of them I didn't come across anything that made me think they weren't.
Muscle/nerve damage and amnesia as side effects of statins It sounds as though memory problems aren't routinely monitored, and there's no way of telling (or at least nothing that gets used) if there's muscle damage until the patient gets into trouble.
This doesn't prove that statins aren't worth the risks.
I read David Freedman's book Wrong a while ago. The article (which I have not read yet) appears to be a condensation of the book. I highlighted a few passages while I read. Here they are.
Quotes about the problem of groupthink from the chapter The Idiocy of Crowds:
…the notion that individuals tend to outthink, outdecide, and outperform groups is so well established that it isn’t even studied much anymore. “The average person certainly believes teamwork trumps individual work, but the evidence says otherwise,” says Natalie Allen, an organizational psychologist at the University of Western Ontario who has studied what she calls the “romance of teams.” …
Nearly four decades of research have exquisitely detailed the ineffectiveness of groups. The problems with collaborative and community thinking have been repeatedly highlighted by a stream of studies, starting with the Yale psychology researcher Irving Janis’s classic examination of “groupthink” back in 1972, which showed how groups could reach terrible decisions that none of the individuals in the group ever would have made on his own. As Janis and many others have shown—and as most of us know all too well—groups are frequently dominated not by people who are most likely to be right but rather by people who are belligerent, persuasive, persistent, manipulative, or forceful. Those who are even mildly adept at getting people to go along with them can quickly form small alliances of viewpoint that may in turn convince others to join in, eventually swaying even those with doubts—most of us don’t want to be the odd man out. …
Once a majority opinion is formed, even highly competent, confident people are reluctant to voice opinions that go against it…
…of the ten deadliest plane crashes in history, cockpit tapes reveal that six of them—killing a total of some 2,400 people—took place with at least one crew member being fully aware of the mistake that was about to bring the plane down but staying mostly quiet because the rest of the crew thought differently. …
Academic, financial, and clinical researchers submit to a pack mentality at least as easily as most sorts of groups or communities. “They go off together in the wrong direction, following one another like any collection of humans,” says Peter Sheridan Dodds, a University of Vermont mathematician who also does work in sociology and biology, among other fields. Herd thinking can keep the community trudging along in one direction for years, resistant to all kinds of contrary evidence …
There is almost no replication in science, and even when there is, the replication is ignored:
Okay, so lousy research can slip past peer review into journals. But surely as soon as other researchers put the published results to the test, the truth will out, right? Possibly—except that the vast majority of published research is never replicated or validated, or if it is, there is no record of it in research journals. All but the most prominent research tends to enter the records and forever persist as apparently legitimate by default. Martinson estimates that more than 95 percent of medical research findings aren’t replicated. No wonder: replication is more or less unfundable, and if someone does it on his own nickel, the results probably won’t come to light. Even a study that fails to replicate a published result, stated Nature in an editorial, “is unlikely ever to be published, or even submitted for publication.” In 2006 Nature reporter Jim Giles dug up the fact that two out of the four stories plastered on the cover of a 2002 issue of the journal—that is, half of the biggest stories in the world of science that week—had failed replication, without all that much notice being taken of it. …
Ioannidis, too, found evidence of the persistence of bad findings. He looked at studies reporting the cardiovascular benefits of vitamin E, anticancer benefits of beta-carotene, and anti-Alzheimer’s benefits of estrogen—important studies that were published in 1993, 1981, and 1996, respectively, and that were each convincingly and prominently refuted in one or more larger studies around 1999, 1994, and 2004, respectively. In 2005, the most recent year Ioannidis checked, half of the researchers who cited the original study of vitamin E did so in the context of accepting the original results, and through 2006 a little more than 60 percent cited the original beta-carotene and estrogen studies, though the results had been solidly refuted—thirteen years earlier in the case of beta-carotene.
Scientists have a substantial motive to fudge research or commit outright fraud:
What could motivate such surprisingly nontrivial apparent levels of dishonesty? The answer turns out to be pretty simple: researchers need to publish impressive findings to keep their careers alive, and some seem unable to come up with those findings via honest work. Bear in mind that researchers who don’t publish well-regarded work typically don’t get tenure and are forced out of their institutions. …
So researchers are pressured to come up with study results that are both interesting and positive. But Ioannidis, among many others, is quick to note a problem: the more surprising, novel, and exciting an idea, the less likely it is to be right.† An idea that seems highly likely to be true, that is utterly plausible, is probably not going to seem exciting—it’s the implausibility that often provides most of the novelty and enthusiasm. …
In other words, researchers are essentially highly incentivized to test exciting ideas that are likely to be wrong—and far more likely to be published. Back in 1989 economists at Harvard and the National Bureau of Economic Research estimated that virtually all published economic papers are wrong, attributing this astoundingly dismal assessment to the effects of publication bias. …
And therein lies the motivation to fudge research. If a study’s results don’t clearly support an interesting hypothesis, a researcher is free to stick it in a file drawer and possibly kiss his career or funding good-bye—or, alternatively, he can imagine salvaging the situation by fabricating data, or by doctoring the way the study is conducted in order to produce more attractive data, or by manipulating the analysis of the data, in order to come out with a dubious positive result.
Whistle-blowing is strongly discouraged:
Baltimore would later publicly apologize to O’Toole for the way she was treated, but she lost her job at MIT. The message sent to young researchers everywhere was clear: the science community isn’t a lot friendlier to whistle-blowers than are police departments and tobacco companies. There’s plenty of evidence that O’Toole’s experience accurately reflects the research culture. In a 1993 study, 53 percent of graduate students said they would expect reprisals for reporting a faculty member’s misconduct, and 26 percent of faculty members said they “probably or definitely” would expect retaliation for reporting a colleague—where retaliation can include blocking promotion or sending unflattering reviews of research to journals and funding committees.20 Only 18 percent of assistant professors stated they “definitely” could report conduct and not expect retaliation.
Publication bias, with the help of incompetence or by itself, can produce the same effect as fraud:
…most researchers can get highly publishable and utterly wrong findings simply as a matter of carelessness or oversight, perhaps goosed by a little bit of subtle gamesmanship, even if unconsciously. …
Now let’s imagine there’s interest among twenty teams of scientists in testing a novel scientific theory, bearing in mind that, as we’ve discussed, most novel theories are likely to be wrong. If they all end up with “significant” findings, then we can reasonably propose that nineteen of them will have failed to confirm the theory, quite correctly, and one team will suffer a data fluke that will have led it to mistakenly conclude its work has confirmed the theory. Guess whose study is most likely to be published?
Peer review has much less value in selecting valid science than is popularly believed:
After the Schön scandal, Nature quoted the Princeton professor, Nobel laureate, and former Bell Labs researcher Philip Anderson as saying, “Nature’s editorial and refereeing policy seems to be influenced by the newsworthiness of the work, not necessarily its quality. And Science seems to be caught up in a similar syndrome.” …
It’s peer review that, more than anything else, [in the popular imagination] is supposed to separate the genuine, reliable science served up in research journals from the apparently frequently junky stuff we get in the mass media. Unfortunately, anecdotal evidence that the peer-review process is an effective way of picking out lousy and even fraudulent work is not encouraging. A panel of researchers and editors assembled to advise Science magazine after the Hwang scandal issued a statement noting that reviewers don’t even look for fraud. …
In fact, it is typically science journalists and other outside observers who imagine peer review to be an assurance of study reliability, and less so scientists. “Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth,” Jennings has written.
By capturing the process of peer review, bad scientists can suppress refutation of their own results:
“Prestigious investigators may suppress via the peer-review process the appearance and dissemination of findings that refute their findings, thus condemning their field to perpetuate false dogma,” states Ioannidis.
In re the chilling effects of status: In Gawande's The Checklist Manifesto, he says that one of the valuable effects of checklists is that they enable low status people (like nurses) to tell their "superiors" that some crucial step has been skipped.
Sorry to nitpick on typos, but the researcher's name is spelled in four different ways in the post: Ionnides, Ioannidis, Ioannidde and Ioannides. I suppose not all can be correct.
I first read Ioannidis' paper several months ago and enjoyed it immensely. The Atlantic article is also good and I hope that it gets some much-needed conversation energized.
A couple of remarks.
I do not like the title. Ioannidis has shown that most published medical research is wrong and the title of his paper why most published research findings are false is misleading to me. Many of the pitfalls which he describes are specific to the limits of experimenting on human subjects which is not transferable across the entire universe of experimental science. Some of the pitfalls are general and all scientists should pay attention to them; it would have been a smoother rhetorical move to reflect this in his title.
The biggest issues amongst my friends and co-workers are diet and neurological nutritional supplements. My experience is that both of these areas are currently a quicksand swamp where it seems like a study of some sort can be found to support many positions that are in contradiction; the most obvious example is the thing with low fat diets, Atkins diets, paleo diets, and the list is far too long to include them all. I have a book Sports Nutrition published by The American Dietetic Association, which is a compilation from Track people, Football people, Swimming people, &c--there are fifty separate articles. The advice is all over the map. The most interesting thing in that book was the people who said that protein powders are useless. I have my own ideas on diet but I find my social life is less complicated if I keep them to myself.
There is some interesting information and (some complete nonsense) in the book by Doctor Daniel Amen Making a good brain great. It may just be blowback to his enterprising attitude, but Amen attracted enough detractor attention to get himself an entry on the quackwatch web page. He supplies medical journal citations for nearly all the claims in his book. Some of his recommended supplements I had heard of before, such as ginko biloba. Some of his recommended supplements I had never heard of, such as coenzyme Q. Amen takes twenty-five brain supplements daily. It was reading his book, and seeing his claims, and seeing his citations, which convinced me of the Ioannidis claim long before I read the Ioannidis paper. It seems impossible that all of the Amen claims and citations could be valid; it seems far more likely that most of the Amen claims and citations are false.
There is one fascinating item in Amen's book which I will share as an aside. He is big on brain scanning (SPECT is his favorite protocol). He reported one research project where they were going to scan a large random sample of normal people. They preselected with a questionnaire to reject addicts, diagnosed mentally-ill, people with family history of Alzheimer's and Parkinson's, people with a memory of severe concussion, &c. Then they winnowed their population again with a preliminary brain scan. Of the first selected set, only ten percent of the people measured normal on their first brain scans and were kept in the study group!
This is an anecdotal data point for the contention that "normal" may be a dubious concept. This may be the single biggest flaw in medical research. People vary widely and they will necessarily vary widely in how they behave under experimental study. The normal distribution may be highly unusual for many practical applications.
If it won't complicate your social life too much here, I'm quite curious about your ideas on diet-- and if you don't want to make them public to LW, could you PM me?
The "textbook stomach" compared to the range of actual stomachs.
I saw that chart a long time ago in an article in a vegetarian magazine. The article said that about 10% of the people who try vegetarianism don't thrive on it. It also said that no one needs more than 4 oz. of meat per day to be healthy.l
A little more about variation-- I've talked with a man who keeps records of his blood tests based on the hope that rejuvenation may be possible, and the theory that if it is, it will be very useful to know what blood chemistry details should be used as a target-- some blood factors (sorry, I didn't ask which ones) vary by a factor of ten among healthy people.
I find the paleo argument pretty compelling by virtue of the logic of the argument--i.e. homo sapiens evolved in an environment which did not have agriculture and grains. Also there is an M.D. blogger whose name escapes me who strongly advises people to avoid all processed grains because they pulse the glucose-insulin cycle in the blood. In March I overhauled my diet. I was eating a croissant every morning, two slices of bread every noon, and two to four cookies every evening. My grain consumption is now less than three slices of bread every four days. I eat cheese and yogurt. So I am not paleo. I do eat a serving of fruit and a serving of raw nuts at every meal.
I may be the most paleo person I know.
The other overhaul I did to my diet is the largest mass of food I am now taking in daily is a big plate of frozen vegetables at noon. I buy the two pound sacks of: 1) peas; 2) corn; 3) string beans; 4) broccoli and cauliflower mix; 5) peas and corn and carrots and string beans and lima beans mix. Choosing which of the sacks to pour from at noon is one of the high points in my typical day. My weight is down; my body mass index is down; my workouts have more pep. I am going to be hanging with this diet for a while.
I like Pollan's formula: eat food, not too much, mostly plants. If somebody put that in the rationality quote thread I would upvote it. Beyond his formula, I do not like Pollan at all. He is not a nutritionist; he is a journalist. When I try and read any of his writing at length my eyes roll.
The best partisan of the paleo philosophy in my mind is the anthropologist David Abram who is part of the "re-wilding movement". I am not anywhere close to a re-wilding movement partisan, but his discussions of forager cultures and what those folks may have to teach us makes Michael Pollan and most of the other more popular figures look like they are serving up extremely thin gruel.
Normal testosterone in males varies by a factor of almost 4:241-827 ng/dL. This sticks in my memory because the last time my doctor did my blood I was at the top of the normal range and I asked him how come I don't look like a gorilla. (The 241-827 does not stick in my memory--I looked it up.) I think my doctor is awesome but his answer to that question was not.
Your textbook stomach link is truly a picture worth a thousand words.
I find the paleo argument pretty compelling by virtue of the logic of the argument--i.e. homo sapiens evolved in an environment which did not have agriculture and grains.
The thing is, though, we have things like milk and alcohol that are not significantly older than grain consumption but we've seen evolutionary effects from them.
It is a point of scholarly contention whether people who retain their lactose tolerance with age were able to displace other people through their superior protein income, or taught milk cultivation to other people, who then had increased prevalence of the lactose toleration gene from that selection pressure. I have no more knowledge than the scholars, but consider the existence of a scholarly controversy evidence for the belief I am disposed against.
There's also significant selection pressure against alcoholism genes among people that have access to alcohol (or just for 'alcohol tolerance,' but the first seems a more robust way to put things) and unsanitary drinking water: the result is that European Americans have dramatically lower rates of alcoholism than Native Americans. (The numbers I've seen compare alcoholism rates, not genes linked to alcoholism; so I suspect this is relevant but am not sure.)
So, it seems likely to me that people with European ancestry, at least, are likely to have spent the last hundred generations or so with a high percentage of grain in their diet, and we've seen that can make adjustments to patterns adopted for the previous thousand generations.
As for paleo? My friends that have tried it have all reported positive effects. My guess is that the main effects are better food and better food discipline. Whenever you make a serious attempt to plan your diet, some things disappear which you did not consider before- and my guess is that makes a huge difference. Extensive research shows that a lot of staples of industrial food- like sucrose, or massive levels of corn- are pretty bad for you. Simply preparing things yourself over having them prepared commercially has shown to dramatically reduce calorie intake; it's easy to go to a restaurant and eat the food that tastes great when you didn't see the two sticks of butter go into the pan. Beyond that, sticking to any sort of plan with food will decrease bad snacking and increase general good habits.
So, my advice is, "don't pick a clearly deficient diet, and don't not pick (i.e. default diet)"- beyond that, it doesn't seem like you can do much besides match to your individual tastes. I think it would take a health benefit of 10 extra years for me to give up bread, since I really like bread. (And I'm expecting to live ~100 more years anyway, and so the duration effect has to be pretty massive to counterbalance the quality of life effect.)
That said, I suspect at some point in the next few years I'll buy a blood glucose monitor and see what sort of effects my diet is having, and if I need to make any changes to prevent diabetes.
So, my advice is, "don't pick a clearly deficient diet, and don't not pick (i.e. default diet)"- beyond that, it doesn't seem like you can do much besides match to your individual tastes. I think it would take a health benefit of 10 extra years for me to give up bread, since I really like bread.
I hear you. Paleo = NO bread. I am not choosing to go that far. Seven months ago I had grains at the base of the food pyramid at ~ 40% of my daily calorie intake. Now it is less than 5% and the idea of eating a foot long subway is not appetizing to me at all right now. I am genetically European and I eat dairy and I consume alcohol; it is interesting and disturbing to observe at first hand people who dairy and alcohol make physically ill because they have some different G-A-T-C sequences deep in the works of the organism.
It is a mystery at this point. I have the latest American Dietetic Association Complete Food and Nutrition Guide. They have the food pyramid with grains at the base just like they showed us all in elementary school. My recent experience is that for my bio makeup that pyramid is not the best guide.
At least some people who are into paleo think that 80% or 90% will give you almost all the benefits.
Paleo diets generally consider corn a grain so you might want to avoid that. Some paleo variants (like the one I'm currently following) are ok with cheese and yogurt in moderation (and butter).
I've often worried about medical research because human trials are necessarily so small. And they're expensive to run, so there seems to be an incentive only to conduct an experiment only when it will win acclaim for the experimenter.
I wonder if medicine needs to be supplemented with things that are not clinical trials -- simulation runs or Brin-style big-but-sloppy survey-data studies. These are not a replacement for medical science, but they are at the very least a sanity check. And sanity checks are cheap and quick, so it's easier to ask researchers to do a sanity check than to spend lots of time and money replicating another clinical trial.
While a small study is more likely to be wrong, there are two corollaries to that. The first is the second point raised - effect size. If (to take the extreme case) you have a study of three people who are all eighty years old and in a persistent vegetative state, and you give them Wundadrug, and they all wake up and suddenly look thirty years old, then you don't need a much larger trial to see that Wundadrug is almost certainly doing something interesting.
More importantly in this context, they're easier to replicate. If a study involves a hundred thousand patients over a thirty year period, then realistically nobody is ever going to check to see if they can get the same results. If, on the other hand, it involves ten patients for two weeks, anyone who is interested enough can try to replicate it with minimal effort.
Related: Reliability of ‘new drug target’ claims called into question:
Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility.
Why do you write "Flaws in mainstream science", if you mean specific parts of science only?
Some other mainstream areas have replication rates of more than 95%.
If something applies to white horses only, I would write "white horses" instead of "horses". Otherwise it might suggest (at least to some readers) that it applies to many, most or even all horses. It is not wrong, but it can be misleading.
I'm not sure it is misleading; the material is obviously focused on health and psychology as the areas I read most in, but the the results I discuss should apply to many areas: the specific problems of no incentives for replication or less than p<0.05 significance are common to all areas or all areas which use NHST statistics, etc. You may like to think that hard sciences like chemistry are exempt... but I get a lot of these citations off a biochemistry blog!
Papers I read are mainly physics papers, especially particle physics. Not replicated results there are so rare that they often get significant attention in the community (Blog article) or even mainstream media (OPERA neutrino speed measurement).
The usual study&publication process for a new particle detector looks like that:
Completely new measurements are just a small fraction of the studies - most results confirm earlier experiments and improve the precision.
Hans-Peter Beck-Bornholdt and Hans-Hermann Dubben are writing books and papers on this problem (most medical research is wrong, see "Is the pope an alien?" Nature 381: 730, 1996) for decades. I've even got a popular science book on probability by them. Sadly most stuff is only available in German.
Curiously, Beck-Bornholdt and Dubben seem (at least, in the two papers of theirs you cited) unaware of the Bayesian solution to the problem they pose. In fact, in "Potential pitfalls...", the abstract concludes with "more care in the use of p-values in analysis and interpretation of clinical data is required", which is a bit like saying that the patient bled to death, so more care in the use of blood-letting is required.
That may just reflect the mid-1990s, though. "Is the Pope an alien?" drew these replies in Nature (vol.382 p.480), only one of which gives the Bayesian solution, and adds that "It is a shame that Bayesian methods are not part of all introductory statistics classes." I don't know if they are now.
(BTW, you need to backslash the parentheses in your DOI URL to make it work with markdown.)
"It is a shame that Bayesian methods are not part of all introductory statistics classes." I don't know if they are now.
I'm a grad student/TA in the statistics department at Iowa State University. Bayes doesn't make it into any of our intro classes outside of maybe introducing Bayes' theorem... but I doubt it. It also doesn't make it into our stat classes for non-major grad students. Some non-stat major grad students take master's level stat courses instead. They introduce some Bayesian material depending on the prof, but not much. There is also a master's level course in Bayesian methods, and new PhD level Bayesian courses in both methods and theory.
Relevant info: our dept. is in the top 10 in the US and so relatively typical, but it is also known for being a bastion of frequentism. Bayesians are making inroads in the department though, but intro classes at Duke, for example, might be much more Bayesian.
Someone even wrote a whole book that demonstrates the misuse and failure in the use of probability: The Astrology File: Scientific Proof of the Link Between Star Signs and Human Behavior
Hmmm. The customer reviews seem to suggest that it wasn't interpreted as "a mock".
I was particularly amused by the list of what customers eventually purchased after looking at the amazon page for this book:
What Do Customers Ultimately Buy After Viewing This Item?
67% buy: The Astrology File: Scientific Proof of the Link Between the Star Signs and Human Behaviour by Gunther Sachs Hardcover
33% buy: Oncogenes, Aneuploidy, and AIDS: A Scientific Life and Times of Peter H. Duesberg by Harvey Bialy Paperback
Yes, I don't know if it is one (I just hope it is :-). But in one of the books by Beck-Bornholdt and Dubben it is interpreted as a mock or at least a good example of how you can prove everything you like if you misuse statistics or your knowledge of probability is sufficiently crippled.
I didn't see any evidence on the page, either in the editorial or customer reviews, that this was intended as a "mock". It looks like a seriously intended book of pseudo-science from the Amazon page.
It is a shame that Bayesian methods are not part of all introductory statistics classes. In this case, Bayesian methods quickly reassure us that the Pope is (probably) not an alien.
The people I've read who gave advice based on Ioannidis article strongly recommended eating paleo.
Incidentally, something like the paleo diet works incredibly well for me. I discovered the right details for me about 4 months ago and have quickly lost weight to a point near my optimal weight and I don't have dizzy or tired spells anymore with blood sugar fluctuations. However, calling it the 'paleo' diet begs a question I don't have enough background in evolutionary biology to ask, but I can pose a related question, because it is more concrete:
My father-in-law is on some kind of fruit diet, and he raves about how healthy fruits are, especially if they are fresh and eaten with the peel. Apples, for example, have all these different healthy things in them.
I wonder why an apple should be healthy. Wouldn't any animal be satisfied with a fruit that just had calories? Enough, in any case, to come back for more and scatter the seeds? Why should an apple -- or anything 'natural' -- be so especially healthy for humans?
I wonder why an apple should be healthy. Wouldn't any animal be satisfied with a fruit that just had calories? Enough, in any case, to come back for more and scatter the seeds? Why should an apple -- or anything 'natural' -- be so especially healthy for humans?
You're missing the other side of the story. Humans evolved to obtain their nutritional needs from those foods that were available in the EEA and this effect is probably more significant than the selection pressure in the other direction (on fruits to be nutritionally beneficial to animals that eat them). Humans are adapted to a diet that includes things that were available to them during the long pre-agricultural evolutionary period.
OK. So reading between the lines somewhat and pushing the argument further, would it similarly/analogously/generally be the case that eating nearly any part of an evolutionarily 'old', highly evolved autotroph would usually be healthy, because they have learned how to make things that are useful for living things to have? (That is, ignoring the part of the argument possibly implying that we might have evolved a dependency upon apples in particular, because that seemed unlikely to me. Though I could be convinced because know it is the case to some degree for oranges and scurvy, for example.)
I don't think that is correct. Much autotroph biomass is indigestible (think tree trunks) and some of the rest is deliberately poisonous. A clever plant doesn't want to be eaten. Furthermore, we ought to eat stuff that has good ingredients - we don't really care whether it makes them itself, or steals them from the manufacturer. And in any case, different living things are built from slightly different kinds of stuff - a lot of autotrophs don't even bother with vitamins A, C, D, and E.
And finally, try to avoid using the terms "evolutionarily old" and "highly evolved". Every living thing you see around you is equally "evolved", since every living thing has the same "evolutionary age" - roughly 3.5 billion years.
Every living thing you see around you is equally "evolved", since every living thing has the same "evolutionary age" - roughly 3.5 billion years.
Surely number of generations matters, too?
I'm not sure that this is at all meaningful because things like selection pressure and mutation rate also arguably matter. If one has a species with lots of generations but an incredibly low mutation rate it isn't going to adapt to an environment as much as another species in the same environment with a higher mutation rate.
Ok, then. Number of generations. Which makes human among the least evolved of all species on the planet. Well, maybe some tortoises are less evolved than us, and maybe elephants and whales and sequoias too, but we certainly have evolved through fewer generations than rats, jellyfish, mosquitoes, sunflowers, earthworms, amoebae, and E. coli.
And finally, try to avoid using the terms "evolutionarily old" and "highly evolved".
OK, corrected, thanks. I definitely see the problem with 'highly evolved'. (For example, a "highly evolved" organism could lose and gain the ability to make a vitamin many times over). I was having trouble separating the ideas of 'older' (as in chronologically first) and 'autotroph' (as in independent). Animals don't dependably make the vitamins they need from plants because they can get these vitamins from plants. Plants, though, couldn't and can't depend upon something else producing them, so they make them on their own.
Much autotroph biomass is indigestible (think tree trunks) and some of the rest is deliberately poisonous.
Yes, of course. It was sloppy of me not to add the qualifier, 'if edible'. Instead, what about the validity of this statement: regardless of its edibility, and the amount and type of toxins that might be present, any plant part would be expected to have as many vitamins as a fruit part? The argument being that plants did not have evolutionary pressure to make their fruits particularly full of vitamins?
Furthermore, we ought to eat stuff that has good ingredients - we don't really care whether it makes them itself, or steals them from the manufacturer.
Agreed. My question is why the former would be healthier. Perhaps because manufacturers sequester more, in greater concentrations?
The argument being that plants did not have evolutionary pressure to make their fruits particularly full of vitamins?
Animals have abilities to detect whether what they eat is nutritious, plants give the animals what they want. That includes things like Vitamin C - though that isn't an essential nutrient for most animals.
As mattnewport mentioned above our ancestors evolved to live on fruits. Most animals can synthesis their own vitamin C. We only lost that ability because our ancestors had so much of it in their diet that they didn't need to synthesize it.
If fruits didn't contain vitamin C, we wouldn't have lost the ability to synthesize it, possibly losing the ability to synthesize something else that they did have.
... what about the validity of this statement: regardless of its edibility, and the amount and type of toxins that might be present, any plant part would be expected to have as many vitamins as a fruit part?
I don't see any reason for thinking it would be true. Any plant part should be expected to have as many vitamins as it needs to do what that part of a plant does. Different vegetables are rich in different vitamins.
The argument being that plants did not have evolutionary pressure to make their fruits particularly full of vitamins?
Fruits are relatively rich in vitamin C because they need antioxidants. Except for the seeds, they are relatively poor in B vitamins because they don't conduct a lot of metabolic activity.
... we don't really care whether it makes them itself, or steals them from the manufacturer.
Agreed. My question is why the former would be healthier.
Who says that it is healthier? Oh, there are certainly arguments against dining too high on the food chain, but I am very uncomfortable with any blanket claim that vegetarianism is healthier than being omnivorous.
Fruits are relatively rich in vitamin C because they need antioxidants.
You mean the fruits themselves need antioxidants? That's interesting! And would explain why fruits are high in antioxidants. What do they need them for?
To keep their (concentrated, moist) sugars from oxidizing, and effectively becoming caramel-like or tar-like, and hence less appetizing to the animals that are going to be tricked into distributing the seeds. Or rather, appetizing for too short a time.
Multinationals put anti-oxidants into junk food to promote shelf-life. Nature puts anti-oxidants into fruit to promote branch-life.
Warning! I might just be making this up. Check more authoritative sources if this info matters to you.
Edit: It occurs to me that the essential oils (fragrances) are also subject to being oxidized into something less savory than the original. Maybe more at risk of oxidizing than the sugars.
I would guess this is not true in general since many things do not want to be eaten and so evolve various defense mechanisms. In turn the organisms that eat them may develop counter-measures that enable them to safely digest their meal despite the defense mechanisms but this will depend on the complex evolutionary history of both organisms. Ruminants are adapted to a quite different diet than humans for example.
The people I've read who gave advice based on Ioannidis article strongly recommended eating paleo.
And they're wrong. Modern diet is a lot more paleo than diet before metabolic syndrome epidemic.
Strong correlation pointing the wrong way is a pretty good evidence for lack of causation.
Let me check this at the site. Quoting:
Here's full list of countries for which data exists for both 1961 and 2007, percents are Paleolithic:Neolithic:Industrial by calories.
...
United States of America - 20.36%:50.72%:28.92% - 20.51%:43.43%:36.06%
That's hard to read because the numbers are all squashed together in a line, so I will rewrite based on the description given:
In 1961, the United States diet was 20.36% Paleolithic, 50.72% Neolithic, and 28.92% Industrial.
In 2007, the United States diet was 20.51% Paleolithic, 43.43% Neolithic, and 36.06% Industrial.
The United States diet is probably what pertains to most readers of this forum, and it certainly is what is usually being talked about in the American media when the American media goes on about the obesity epidemic.
But the modern United States diet is not "a lot more Paleo". That's a wildly wrong summary of what the data shows. Rather, the modern diet is a lot more Industrial. Specifically, the modern diet has a lot more sweeteners and vegetable oils.
The summary "they're wrong...modern diet is a lot more paleo..." creates the impression that we have largely taken paleo advice and have suffered greatly for it. But in the case of the United States, where the obesity epidemic is massive, that is simply not the case. The big change is in consuming more sweeteners and more vegetable oils - according to the data presented.
Moreover, my impression - confirmed when googling this - is that the warning against sweeteners and vegetable oils is an important part of the paleo critique of the modern diet. If you follow paleo advice, you will abandon sweeteners and vegetable oils.
So, the correlation does not "go the wrong way", in the case of the United States. There are a lot of other countries listed there, but since I don't know whether and how much they are subject to an obesity epidemic, I can't make use of the data.
But the modern United States diet is not "a lot more Paleo".
You're wrong. 1961 USA was already quite far in its transition away from Neolithic food, it just takes time for such low-level damage to accumulate.
There's little high quality world-wide data before 1961, but feel free to offer a bet about it if you doubt what it would reach.
Correlation goes the wrong way, just like I said.
In any case, obesity statistics are easily available for all countries, so I'm puzzled by your unwillingness to take a look.
Moreover, my impression - confirmed when googling this - is that the warning against sweeteners and vegetable oils is an important part of the paleo critique of the modern diet. If you follow paleo advice, you will abandon sweeteners and vegetable oils.
It's not. Paleos bundle all non-Paleo foods as if they were pretty much the same. It's about as useful as abstinence-only approach to safe sex.
No, paleo dieters don't bundle all non-paleo foods. Almost all foods that can be bought in a store or farmers market has been altered in one way or another since the neolithic began. And most paleo dieters acknowledge this. The purpose is not to perfectly replicate the diet of a paleolithic human, in the fashion of a civil war reenactor, but to most closely mimic the profile of the diet humans evolved to eat.
Some factors in this are nutrient density, lower simple carbs, intermittent fasting, and ketosis. These have been shown in research to be beneficial. But of course, there is a lot of research that shows a diversity of results. The strongest evidence comes from studies of hunter-gatherers who mostly lack chronic diseases, and this forms a central basis of the paleo diet.
By the way, Americans ate more meat than bread prior to the 20th century. They also only ate a fraction of the sugar in the past. Yet many of the chronic diseases only became common in the 20th century. Heart disease was a rare disease in the early 20th century, but by the end of the century it was rampant.
In a recent article, John Ioannidis describes a very high proportion of medical research as wrong.
Part of the problem is that surprising results get more interest, and surprising results are more likely to be wrong. (I'm not dead certain of this-- if the baseline beliefs are highly likely to be wrong, surprising beliefs become somewhat less likely to be wrong.) Replication is boring. Failure to replicate a bright shiny surprising belief is boring. A tremendous amount isn't checked, and that's before you start considering that a lot of medical research is funded by companies that want to sell something.
Ioannidis' corollaries:
The culture at LW shows a lot of reliance on small inferential psychological studies-- for example that doing a good deed leads to worse behavior later. Please watch out for that.
A smidgen of good news: Failure to Replicate, a website about failures to replicate psychological findings. I think this could be very valuable, and if you agree, please boost the signal by posting it elsewhere.
From Failure to Replicate's author-- A problem with metastudies:
The people I've read who gave advice based on Ioannidis article strongly recommended eating paleo. I don't think this is awful advice in the sense that a number of people seem to actually feel better following it, and I haven't heard of disasters resulting from eating paleo. However, I don't know that it's a general solution to the problems of living with a medical system which does necessary work some of the time, but also is wildly inaccurate and sometimes destructive.
The following advice is has a pure base of anecdote, but at least I've heard a lot of them from people with ongoing medical problems. (Double meaning intended.)
Before you use prescription drugs and/or medical procedures, make sure there's something wrong with you. Keep an eye out for side effects and the results of combined medicines. Check for evidence that whatever you're thinking about doing actually helps. Be careful with statins-- they can cause reversible memory problems and permanent muscle weakness. Choose a doctor who listens to you.
Forum about self-experimentation-- note: even Seth Roberts is apt to oversell his results as applying to everyone.
Link about the failure to replicate site found here.