bentarm comments on How to prove anything with a review article - Less Wrong

17 Post author: RichardKennaway 22 November 2011 12:04PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (11)

You are viewing a single comment's thread.

Comment author: bentarm 23 November 2011 09:56:03AM *  6 points [-]

Proper quacks even announce that they're cherry-picking in the title... An Overview of Positive Homeopathy Research (PDF).

As far as I can tell, there is a strong trend, at least in the medical sciences towards more rigorous meta-analyses (eg, the Cochrane Reviews) which to a large extent avoid this problem. I'm not sure to what extent meta-analyses of this type are the norm in other sciences.

Comment author: jimmy 23 November 2011 09:00:47PM 3 points [-]

I think homeopathy is as silly as you do, but there are reasons to look at only the positive results, and the first step to doing it right is acknowledging what you're doing, so I don't fault them for the title.

Consider cases where there are a lot of unknowns on how to do it right. Most studies will find no effect, which is very good evidence that the average researcher can't make it work and you shouldn't just ask a random researcher to can treat you. However, if you look at the distribution, you might find that there's a false positive distribution and a second - an interesting distribution of people that it does consistently work for. Then you can go on to study this separate distribution and see what they're doing right.