Why we need better science, example #6,281

lukeprog

Avorn (2004) reports:

In a former British colony, most healers believed the conventional wisdom that a distillation of fluids extracted from the urine of horses, if dried to a powder and fed to aging women, could act as a general tonic, preserve youth, and ward of a variety of diseases. The preparation became enormously popular throughout the culture, and was widely used by older women in all strata of society. Many years later modern scientific studies revealed that long-term ingestion of the horse-urine extract was useless for most of its intended purposes, and that it causes tumors, blood clots, heart disease, and perhaps brain damage.

The former colony is the United States; the time is now; the drug is the family of hormone replacement products that include Prempro and Premarin (manufactured from pregnant mares' urine, hence its name). For decades, estrogen replacement in postmenopausal women was widely believed to have "cardio-protective" properties; other papers in respected medical journals reported that the drugs could treat depression and incontinence, as well as prevent Alzheimer's disease. The first large, well-conducted, controlled clinical trial of this treatment in women was not published until 1998: it found that estrogen replacement actually increased the rate of heart attacks in the patients studied. Another clinical trial published in 2002 presented further evidence that these products increased the risk of heart disease, stroke, and cancer. Further reports a year later found that rather than preventing Alzheimer's disease, the drugs appeared to double the risk of becoming senile.

Armstrong (2006) adds:

The treatment seemed to work because those who used the drug tended to be healthier than those who did not. This was because it was used by people who were more interested in taking care of their health.

Avorn (2004) reports:

In a former British colony, most healers believed the conventional wisdom that a distillation of fluids extracted from the urine of horses, if dried to a powder and fed to aging women, could act as a general tonic, preserve youth, and ward of a variety of diseases. The preparation became enormously popular throughout the culture, and was widely used by older women in all strata of society. Many years later modern scientific studies revealed that long-term ingestion of the horse-urine extract was useless for most of its intended purposes, and that it causes tumors, blood clots, heart disease, and perhaps brain damage.

The former colony is the United States; the time is now; the drug is the family of hormone replacement products that include Prempro and Premarin (manufactured from pregnant mares' urine, hence its name). For decades, estrogen replacement in postmenopausal women was widely believed to have "cardio-protective" properties; other papers in respected medical journals reported that the drugs could treat depression and incontinence, as well as prevent Alzheimer's disease. The first large, well-conducted, controlled clinical trial of this treatment in women was not published until 1998: it found that estrogen replacement actually increased the rate of heart attacks in the patients studied. Another clinical trial published in 2002 presented further evidence that these products increased the risk of heart disease, stroke, and cancer. Further reports a year later found that rather than preventing Alzheimer's disease, the drugs appeared to double the risk of becoming senile.

Armstrong (2006) adds:

The treatment seemed to work because those who used the drug tended to be healthier than those who did not. This was because it was used by people who were more interested in taking care of their health.

Absent any other prior, why would you use anything other than "My body will react to hormones the same way most other people's bodies react to hormones"?

First, because I am not absent other informational priors. I have a lifetime of informational priors about my own body. I also have access to pubmed, wikipedia, my 23andme genomic data, my personal medical history, my family's medical history, and lab testing services that can take accurate measurements of me.

There are no clinical trials that have controlled for that information.

Second, because I know others are not identical to me. Basing my choices solely on some statistical outcome on a pool of patients where I have none of that kind of information, and indeed the doctors involved didn't take that information, and didn't factor it into their solutions for their patients, strikes me as throwing out most all of my relevant data and trusting the results produced by a blind man with a shotgun.

Moreover, refusing to experiment on yourself is to refuse to look at reality and take actual data about the system you're interested in - you. That's poor decisions theory, poor inference, and poor problem solving.

Yes, medicine is weird. Therefore, instead of thinking that you have it all worked out, or that a clinical trial has it all worked out for you, the rational thing to do is to evaluate options that might work, their costs and risks, try things, take measurements, update your model based on that additional data, and try again. Sure, if there are clinical trials, avail yourself of that information as well. Nice place to find candidate treatments. But you're deluded if you think a positive result means it will assuredly work for you, and you're deluded if you think a negative results, a "failure to reject", means it won't. At a minimum, if the trial didn't have a crossover study, it hasn't ruled out that the treatment is a perfect cure for some subset of people with the problem.

Any decent doctor I've had has basically said that all treatments are experiments for a particular person - maybe it will work for you, maybe not.

I don't know that I have incentives any different from anyone else with a malady. I wish to get better. I recognize that there are risks involved in the attempts to get better. What doctors fail to appreciate, probably because it's not really their problem, is that doing nothing also has a cost - the likely continuance of my malady.

We don't limit our pool of potential solutions to our problems to solutions "validated" by double blinded placebo controlled trials in any other aspect of life, because it isn't rational to do so. It's not rational for medical problems either.

We don't limit our pool of potential solutions to our problems to solutions "validated" by double blinded placebo controlled trials in any other aspect of life, because it isn't rational to do so.

Wrong. We don't do it because either there are no publications that answer our questions, so that we have to use something else, or because we have a more convenient method that works. Please don't appeal to "rationality".

0Vladimir_Nesov14y

Information is relevant only to the extent you can use it. How specifically can you use it to improve on prior provided by studies, and why would that modified estimate be an improvement? (Every improvement is a change, but not every change is an improvement.)

12Scott Alexander14y

It's not about a positive result meaning something will "assuredly work for you". Only a Sith deals in absolutes. It's about cost-benefit analysis. To give an example, no reasonable person would self-experiment to see if cyanide cures their rash. Although there's a distant probability your body has some wildly unusual reaction to cyanide in which it cures rashes, it's much more likely that cyanide will kill you, the same way it kills everyone else. Although it might be worth a shot if cyanide had no downside, we have very strong evidence that on average it has a very large downside. The same is true of HRT. People were using it to improve their cardiovascular health. We found that, on average, it decreases cardiovascular health. You can still try using it on the grounds that it might paradoxically increase yours, but on average, you will lose utility. Consider the analogy to a lottery. You have different numbers than everyone else does. Just because someone else lost the lottery with their numbers, doesn't mean you will lose the lottery with your numbers. But if we study all lottery participants for ten years and find that on average they lose money, then unless you have a specific reason to think your numbers are better than everyone else's (not just different), you should expect to lose money too. Now things would be different with a treatment with no downside (like eating a lot of some kind of food, or taking a safe and cheap supplement) - as long as you don't mind the loss of time and money you can experiment all you want with those (though I still think you'd have trouble with bias and privileging the hypothesis, and that a rational person wouldn't find a lot of these harmless self-experiments worth the time and the money at all). And things would be different if the potential benefit and potential harm had different levels of utility for you: for example, if you wanted to cure your joint pain so badly you didn't mind risking heart attack as a side effect.

43

Why we need better science, example #6,281

43

43

43

Why we need better science, example #6,281

43

43