army1987 comments on Too good to be true - Less Wrong

24 Post author: PhilGoetz 11 July 2014 08:16PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (119)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 22 July 2014 01:36:50AM *  3 points [-]

Publication in general doesn't have to be rare and special, only the publications of negative results has to be uncommon.

I realize that, and I've already pointed out why the difference in rates is not going to be that large & that your cite does not explain the excess significance in their sample.

There's other issues, e.g. how many of those tests were re-testing simple, effective FDA-approved drugs and such?

Doesn't matter that much. Power, usually quite low, sets the upper limit to how many of the results should have been positive even if we assume every single one was testing a known-efficacious drug (which hypothesis raises its own problems: how is that consistent with your claims about the language bias towards publishing cool new results?)

Also, for the Soviet union, there would be a certain political advantage in finding no efficacy of drugs that are expensive to manufacture or import.

So? I don't care why the Russian literature is biased, just that it is.

What's of the chemical safety studies? There's a very strong bias to fail to disprove the null hypothesis.

Yes, but toxicology studies being done by industry is not aimed at academic publication, and the ones aimed at academic publication have the usual incentives to find something and so are part of the overall problem.

Yet your paper somehow found a ridiculously high positive rate for acupuncture. The way I think it would work, well, first thing first it's very difficult to blind acupuncture studies and inadequately blinded experiments should find positive result from the placebo effect,

Huh? The paper finds that acupuncture study rates vary by region. USA/Sweden/Germany 53/59%/63%, China/Japan/Taiwan 100% etc

secondarily, because that's the case, nobody really cares about that effect, and thirdly, de-facto the system did not result in construction of acupuncture centres.

How much have you looked? There's plenty of acupuncture centres in the USA despite a relatively low acupuncture success rate.

I haven't really noticed nootropics being a big thing

Does a fish notice water? But fine, maybe you don't, feel free to supply your own example of Russian pseudoscience and traditional medicine. I doubt Russian science is a shining jewel of perfection with no faults given its 91% acupuncture success rate (admittedly on a small base).

but somehow they didn't end up replacing antibiotics with homebrew phage therapy

Not sure that's a good example, as Wikipedia seems to disagree about homebrew phage therapy not being applied: https://en.wikipedia.org/wiki/Phage_therapy#History

When antibiotics were discovered in 1941 and marketed widely in the U.S. and Europe, Western scientists mostly lost interest in further use and study of phage therapy for some time.[12] Isolated from Western advances in antibiotic production in the 1940s, Russian scientists continued to develop already successful phage therapy to treat the wounds of soldiers in field hospitals. During World War II, the Soviet Union used bacteriophages to treat many soldiers infected with various bacterial diseases e.g. dysentery and gangrene. Russian researchers continued to develop and to refine their treatments and to publish their research and results. However, due to the scientific barriers of the Cold War, this knowledge was not translated and did not proliferate across the world.

Anyway,

To summarize, I see this allegation of some grave fault but I fail to see the consequences of this fault.

How do you see the unseen? Unless someone has done a large definitive RCT, how does one ever prove that a result was bogus? Nobody is ever going to take the time and resources to refute those shitty animal experiments with a much better experiment. Most scientific findings never gets that sort of black-and-white refutation, it just gets quietly forgotten and buried, and even the specialists don't know about it. Most bad science doesn't look like Lysenko. Or look at evidence-based medicine in the West: rubbish medicine doesn't look like a crazy doc slicing open patients with a scalpel, it just looks like regular old medicine which 'somehow' turns up no benefit when rigorously tested and is quietly dropped from the medical textbooks.

To diagnose bad science, you need to look at overall metrics and indirect measures - like excess significance. Like 91% of acupuncture studies working.

Comment author: [deleted] 25 July 2014 11:32:00AM 1 point [-]

Does a fish notice water?

Well, humans do notice air some of the time. (SCNR.)