John_Maxwell_IV comments on Case study: Melatonin - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (172)
I'm confused... did you update at all on the possibility of harms from long-term melatonin use after reading the article?
No. Your excerpt was a poor exposition of the standard precautionary principle I regard as entirely useless.
Reading your link now, I have even less reason to pay attention to it. It's a random press release about a presumably small unreplicated animal study in a species I don't know to be particularly germane to humans (eg. chimps) about changes of unclear importance in a body system with no human analogues ("In birds, switching off GnRH causes the gonads -- testes and ovary -- to shrink as part of the birds' yearly cycle.") with doses potentially high enough to be completely irrelevant to human supplementation (injecting melatonin?). I haven't even read the study!
Combine all the conditionals here (the smallness and lack of replication alone knocks down the chance this means anything about anything to well under 50%), and I don't see why I would update at all (not being an AI or anything which can represent degrees of belief with 64-bit floats).
If anything, I think this sort of study is a good example of why animal studies should be ignored in discussing supplements.
Good points.
Have there been any meta-analyses of how well supplement studies on animals tend to transfer to humans?
On the general topic of animal model external validity & translation to humans (with obvious relevance to supplements & nootropics), here are the major systematic reviews, meta-analyses, and articles criticizing the routine failures of animal models to provide any meaningful information about dangers or benefits in humans, and documenting the even lower quality of animal experiments than usual in (human) medicine or psychology:
I spent the day reading up on the topic. The transfer or replication rates range from around 40% on the upper end to zero in some cases involving as many as hundreds of attempts to transfer. The methodological quality of the animal studies are usually terrible with hardly any blinding and randomization rare (and when it is done, one author indicates that researchers surveyed would say it was done as informally as grabbing random mice/rats out of the cages), and the publication biases at play seem to be even larger than in human studies.
(Reading the papers, I found myself disturbed by the ethical implications: when human studies fall to publication bias, 'all' that does is waste many millions of dollars and impede the progress of science and substantially inconvenience many subjects and put them at risk; but when at least a third of all animal experiments never get written up, and half of chimpanzee medical/biological studies never get cited when they do get published, that means that the billions of animals gone through every few years are mutilated and tortured and killed for absolutely nothing at all. This makes me much more sympathetic to the NIH's recent retirement of its research chimpanzees.)
There are some interesting examples, though; from one of the links:
From the toxicology paper:
From my appendix:
(I imagine that the latter is particularly relevant; how many thousands of mouse studies on inflammation were part of the evidence base for those <150 clinical trials? Probably quite a few. And it seems that they all are essentially irrelevant to anything in humans. Now, imagine the translation rate for bird to primate, based on a bird system which doesn't even exist in humans...)
Thanks for doing all this research! You should make a page on gwern.net, you're wasting your talents going in to this kind of depth in an ancient comments thread ;)
Perish the thought of there being any waste!
I suspect it might also depend, among other things, on how closely related those animals are to us. I would likely take your study more seriously if it was done on mammals, and even more so if it was done on primates.