Cyan comments on Beautiful Probability - Less Wrong

34 Post author: Eliezer_Yudkowsky 14 January 2008 07:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (109)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: VAuroch 02 September 2015 12:27:40AM 1 point [-]

Double-blind trials aren't the gold standard, they're the best available standard. They still don't replicate far too often, because they don't remove bias (and I'm not just referring to publication bias). Which is why, when considering how to interpret a study, you look at the history of what scientific positions the experimenter has supported in the past, and then update away from that to compensate for bias which you have good reason to think will show up in their data.

In the example, past results suggest that, even if the trial was double-blind, someone who is committed to achieving a good result for the treatment will get more favorable data than some other experimenter with no involvement.

And that's on top of the trivial fact that someone with an interest in getting a successful trial is more likely to use a directionally-slanted stopping rule if they have doubts about the efficacy than if they are confident it will work, which is not explicitly relevant in Eliezer's example.

Comment author: Cyan 02 September 2015 09:28:16PM 0 points [-]

I can't say I disagree.