Emily comments on Original Research on Less Wrong - Less Wrong

21 Post author: lukeprog 29 October 2012 10:50PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (47)

You are viewing a single comment's thread. Show more comments above.

Comment author: Emily 03 November 2012 03:37:27PM 1 point [-]

it should only take <2.5 times as many subjects to detect a real effect at p < .001 instead of p < .05

Not to disagree with the overarching point, but the use of "only" here is inappropriate under some circumstances. Eg, a neuropsychological study requiring participants with a particular kind of brain injury is going to find more than doubling its n extremely difficult and time-consuming. For this kind of study (presuming insistence on working with p-values) it seems better to roll with the "ludicrous" p < .05 and rely on replication elsewhere for improved reliability. "Ludicrous" is too strong in fields with small effect sizes and small subject pools; they just need a much higher rate of replication.