Eliezer_Yudkowsky comments on Outside the Laboratory - Less Wrong

63 Post author: Eliezer_Yudkowsky 21 January 2007 03:46AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (336)
Tags: religion antitheism lawful_rationality lawful_universe

Comments (336)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: ChristianKl 26 December 2012 03:24:28PM 0 points [-]

Sorry, ambiguous wording. 0.05 is too weak, and should be replaced with, say, 0.005. It would be a better scientific investment to do fewer studies with twice as many subjects and have nearly all the reported results be replicable.

I'd rather prefer two studies with 0.05% on the same claim by different scientifists to one study with 0.005%. Proving replicable of scientific studies with actually replicating them is better than going for a even lower p value.

Comment author: gwern 26 December 2012 08:06:49PM *  3 points [-]

I'd rather prefer two studies with 0.05% on the same claim by different scientifists to one study with 0.005%.

I wouldn't. Two studies opens the door to publication bias concerns and muddles the 'replication': rarely do people do a straight replication.

From Nickerson in http://lesswrong.com/lw/g13/against_nhst/

Experiments that are literal replications of previously published experiments are very seldom published - I do not believe I have ever seen one. Others who have done systematic searches for examples of them confirm that they are rare (Mahoney, 1976; Sterling, 1959)....PhD committees generally expect more from dissertations than the replication of someone else's findings. Evidence suggests that manuscripts that report only replication experiments are likely to get negative reactions from journal reviewers and editors alike (Neuliep & Crandall, 1990, 1993)

Comment author: Eliezer_Yudkowsky 27 December 2012 02:20:00AM 2 points [-]

I wouldn't. Two studies opens the door to publication bias concerns

Agreed. It's much easier for a false effect to garner two 'statistically significant' studies with p < .05 than to gain one statistically significant study with p < .005 (though you really want p < .0001).

Powered by Reddit Powered by Reddit