Here is what Eddy has found in his research. Give a group of cardiologists high-quality coronary angiograms (a type of radiograph or x-ray) of typical patients and they will disagree about the diagnosis for about half of the patients. They will disagree with themselves on two successive readings of the same angiograms up to one-third of the time. Ask a group of experts to estimate the effect of colon-cancer screening on colon-cancer mortality and answers will range from five percent to 95 percent.
An excerpt. Scary!
Unfortunately, Scientific American doesn't give any clues as to what study this information comes from, and my attempts to find it have all come up blank (anyone else have more luck?)
Of the two studies I can find on inter-rater reliability of coronary angiography, one shows greater than 98%, and the other shows high 90s. No doubt these are completely different types of coronary scans with completely different criteria for success than whatever Eddy was doing, but given the lack of pointers to the original study it's impossible to say exactly what's going on.
A feature in Scientific American magazine casts some light on the troubled state of modern medicine.
Health Care Myth Busters: Is There a High Degree of Scientific Certainty in Modern Medicine?
Short excerpt:
Scientific American often gates its online articles after some time has passed, so I don't know how long it will be available.