Comment Permalink

wedrifid14y100

Here is what Eddy has found in his research. Give a group of cardiologists high-quality coronary angiograms (a type of radiograph or x-ray) of typical patients and they will disagree about the diagnosis for about half of the patients. They will disagree with themselves on two successive readings of the same angiograms up to one-third of the time. Ask a group of experts to estimate the effect of colon-cancer screening on colon-cancer mortality and answers will range from five percent to 95 percent.

An excerpt. Scary!

Scott Alexander14y140

Unfortunately, Scientific American doesn't give any clues as to what study this information comes from, and my attempts to find it have all come up blank (anyone else have more luck?)

Of the two studies I can find on inter-rater reliability of coronary angiography, one shows greater than 98%, and the other shows high 90s. No doubt these are completely different types of coronary scans with completely different criteria for success than whatever Eddy was doing, but given the lack of pointers to the original study it's impossible to say exactly what's going on.

2SilasBarta14y

Double-scary when you think about how much education one must (legally) get before becoming a cardiologist. There's a huge amount of waste somewhere in all that.

See in context

13 Link: "Health Care Myth Busters: Is There a High Degree of Scientific Certainty in Modern Medicine?"

by CronoDAS

1st Apr 2011

1 min read

13

A feature in Scientific American magazine casts some light on the troubled state of modern medicine.

Health Care Myth Busters: Is There a High Degree of Scientific Certainty in Modern Medicine?

Short excerpt:

We could accurately say, "Half of what physicians do is wrong," or "Less than 20 percent of what physicians do has solid research to support it." Although these claims sound absurd, they are solidly supported by research that is largely agreed upon by experts.

Scientific American often gates its online articles after some time has passed, so I don't know how long it will be available.

Personal Blog

13

New Comment

34 comments, sorted by

top scoring

Click to highlight new comments since: Today at 11:11 AM

[-]Scott Alexander14y210

Well, the good news is that between Robin Hanson's claim that we receive twice as much health care as we should and these guys' claim that "Americans only receive about half of recommended medical care", on average we're getting it pretty darned close!

[-]Daniel_Burfoot14y80

Right. Also, since most published research findings are wrong anyway, it's not such a big deal that doctors don't know about them! Life is good!

[-]wedrifid14y100

Here is what Eddy has found in his research. Give a group of cardiologists high-quality coronary angiograms (a type of radiograph or x-ray) of typical patients and they will disagree about the diagnosis for about half of the patients. They will disagree with themselves on two successive readings of the same angiograms up to one-third of the time. Ask a group of experts to estimate the effect of colon-cancer screening on colon-cancer mortality and answers will range from five percent to 95 percent.

An excerpt. Scary!

[-]Scott Alexander14y140

Unfortunately, Scientific American doesn't give any clues as to what study this information comes from, and my attempts to find it have all come up blank (anyone else have more luck?)

[-]SilasBarta14y20

Double-scary when you think about how much education one must (legally) get before becoming a cardiologist.

There's a huge amount of waste somewhere in all that.

[-]jimrandomh14y70

As I understand it, medical school is very heavy on memorization - doctors are expected to absorb a huge quantity of data about procedures, medications, etc. So much memorization, in fact, that it's probably incompatible with a habit of model-checking everything. If that's true, then this would filter out the traits that medicine needs most to repair its practices.

[-]Raemon14y50

Or the field really is that confusing, which wouldn't surprise me too much. You're dealing with variables that are constantly changing.

[-]SilasBarta14y10

I'm pretty sure it doeesn't take 4 years of "whatever you feel like majoring in" in university before even starting med school, just to learn an angiogram classifier that a computer can probably outperform you on anyway.

At least, the English lit classes could probably be scrapped ...

[-]Raemon14y00

When it comes to medicine, if computers CAN outperform the average person, we should probably be using computers anyway. (Yeah, that maxim applies to most professions, but most professions don't have lives on the line).

[-]SilasBarta14y20

When it comes to medicine, if computers CAN outperform the average person, we should probably be using computers anyway.

Hey, if you want to get doctors to step out of the way on the grounds that a computer really can trounce their expert judgment, even in just a few domains ... well, you're going up against a lot of resistance.

[-]Raemon14y30

Well, yeah. That will definitely be an issue, if not now then in a few years. But I also wouldn't be surprised if this was a genuinely difficult task, and I don't know that statements like "a computer can probably outperform you" are justifiable to throw around, unless you actually know that computers DO have a better track record at the task in question.

[-]SilasBarta14y10

I hadn't read it myself, but I remember that there are a lot of stories like these in Ian Ayers's Supercrunchers and the success of simple algorithms over expert judgment and the resistance thereto. And Superfreakonomics mentioned the story of how hard it was to get doctors to wash their hands as often as necessary.

[-]EchoingHorror14y30

Hurray! A portion of what physicians do has solid research to support it! There are ways that work; we just have to do those more often and other things not at all.

What if the 15% of doctors who can at least follow the numbers started a medical sub-community dedicated to making decisions based on solid research and standardizing the practice of medicine, then commissioned an independent study of their success rate? That could take objective models for treatment, spread them to those who understand how they work, then spread them to everyone else who can see them working better than an average physicians choices.

Individual doctors could be certified by their sub-community as users of the research-supported ways, and their value would increase, and more would join them.

[-]Bo10201014y30

Would a patient thus do better to research his or her symptoms online before going to the doctor's office, and then insisting on the treatment provided?

If so is there a good place to do this research? Are there good websites that are usually informed with up-to-date research on a variety of topics? I haven't had a health issue that's needed such research in recent memory, but if I did I would probably type my symptoms into Google alongside technical sounding words like "incidence," "epidemiology," and "differential diagnosis."

[-]Scott Alexander14y80

Symptom checker programs and Google are good preliminary resources, especially if you want to figure out whether it's worth your time going to the doctor or not for something mild.

If you Google your symptoms, then go to the doctor and insist you have whatever disease came up on top, and tell the doctor ey's wrong if ey disagrees or wants to do further tests, then you become the kind of patient who doctors tell hilarious stories about in hospital cafeterias. These stories are rarely hilarious for the patient involved unless they have a very masochistic sense of humor.

The more honest symptom checkers will give you a list of many possible conditions that could be causing your symptoms. Unless you have something really obvious, most doctors won't be able to do better than a list until they've done tests. Symptom checkers usually don't adjust for your age, your past history, your current medications, or many other things that are very important for serious illnesses, and they can only give a very probabilistic result. They have their place but in a serious condition are no replacement for a doctor.

[-]saturn14y10

On the other hand, without some kind of prompting, many doctors tend to narrowly focus on reaching a firm diagnosis at the expense of maximizing benefit/minimizing risk to the patient.

[-]Scott Alexander14y70

There are certainly different schools of thought possible on how much time to invest in disease identification before going for a treatment, but can you explain your evidence for why you think most doctors tend to err on the side of over-caution?

Medicine does include the ideas of "empirical treatment" and "empirical diagnosis". Empirical treatment is when eg a doctor can't figure out exactly what a disease is, but it looks bacterial, so ey'll throw some common antibiotics at it and see if it works. Empirical diagnosis is when a doctor isn't sure about a particular diagnosis, so ey gives the treatment for that diagnosis and then if the patient gets a bit better ey's convinced and moves on to more serious long-term treatment.

These are useful but still not as good as knowing what you're doing. First of all, a lot of the time they complicate the picture enough to make real diagnosis impossible; an organism that could have been cultured and identified before starting empirical antibiotics might be decimated enough to become unidentifiable afterwards, but not so dead that it can't bounce back and cause a relapse, so starting empirical treatment can be dangerous if you think you're ever going to change your mind and want to figure out exactly what's wrong.

And a lot of time, the treatment for one disease will make another similar disease worse. Steroids are the best treatment for a lot of auto-immune conditions, but they will make infectious conditions worse; immune conditions can look like infectious conditions if you haven't investigated them properly; this is especially true in rashes, which some doctors prescribe steroids for almost automatically. With rashes it's not the end of the world because even if a rash gets worse you probably won't die from it, but it still annoys patients and there are some other conditions where you don't have that margin of error.

There's also just the common sense problem of people getting really angry if they've been told they have a disease, they've changed their life around to help live with that disease, they've been taking medicine for that disease which may have all sorts of side effects, and then they go to a second doctor and that doctor tells them they don't in fact have the disease. I saw a lady the other day who for eleven years thought she had multiple sclerosis, switched neurologists, and then the new neurologist told her she didn't have it - that's an extreme and very rare example and I have a feeling the error is with the new guy rather than the old guy, but even if so that level of uncertainty and confusion is still traumatic and is a good reason for doctors to work really hard to get a firm diagnosis before giving it.

Maybe I'm missing your point; if so can you give an example of what you mean?

[-]Emile14y00

Symptom checker programs

Any recommendations for those? Google returns a lot of results, and I don't really know of a good way to filter between them. I didn't even know such programs existed!

Hmm, iTriage for Android looks like it has good reviews.

[-]jimrandomh14y10

If you're willing and able to read medical research directly, search PubMed: www.ncbi.nlm.nih.gov/pubmed.

[-]CronoDAS14y00

I don't know how good it actually is, but wrongdiagnosis.com appears to be trying to do this...

[-]knb14y00

The interesting thing is that, although Doctors seem to add little net value, their pay is very high.

[+]Manfred14y-70

[-]drethelin14y00

Is there an established procedure for fixing an entire profession? What's the solution if almost all doctors are wrong about a great many things? I suppose changing Med School to produce new more competent doctors could work, except then every new doctor has to work with old doctors as soon as they graduate that do things in a different and worse way.

[-]TheOtherDave14y60

This article, about the process and results of incorporating standard checklists into medical procedures, seems like a reasonable example of a success story and makes me wonder how many improvements could be implemented, not so much by making more competent doctors, but by pushing the cultural idea that doctors, like programmers, are technical specialists who operate best as part of a team that includes competent project managers.

[-]DanielLC14y00

Start a business that does it right. If you can get past the regulators and such, and there are enough rational customers, you'd do better than everyone else, and they'd start copying you.

[-]Vladimir_M14y30

Start a business that does it right. If you can get past the regulators and such, and there are enough rational customers, you'd do better than everyone else, and they'd start copying you.

Well, good luck getting past regulators in a profession that operates as a self-regulating guild!

In my opinion, the only mechanism of competition that could conceivably ameliorate the systematic problems of medicine is the international medical tourism. However, I'm far from certain that free competition has much potential for preventing medicine from drifting away from reality, considering how much people are prone to biased and even outright magical thinking on this subject, even when all the incentives to get things right are in place.

[-]DanielLC14y00

I think business have to be more rational than people in general. If you pay for your own health care, this would be a problem, but if you have health insurance, they'll just pay for what works.

[-]Eugine_Nier14y10

If you pay for your own health care, this would be a problem, but if you have health insurance, they'll just pay for what works.

This is more-or-less the reason health insurance companies are so hated.

[-][anonymous]14y00

Not necessarily. The whole rather absurd U.S. institution of employer-paid health insurance is an artifact of various government-imposed incentives whose roots range back to WW2-era wage controls. The employers are artificially incentivized to pay a part of the wages in medical benefits instead of cash -- and if the employees value spurious medicine, there is no good reason for the employers to argue with their misconceptions. They'd happily pay part of the wages in astrological benefits if there was a comparable demand and regulatory incentive structure.

Of course, the employers would like to limit their overall expenses on medical benefits, but it's far from clear that questioning the effectiveness of medicine is a practical way to do so, especially since it's a strong social taboo. The idea of philistine and profiteering capitalists questioning the opinion of the wise and noble medical profession sounds blasphemous to the modern public opinion, and this could result in many bad consequences for the former.

[-]nazgulnarsil14y00

start over in a new country. over time health outcomes will drive immigration.

Moderation Log