You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

gwern comments on Value of Information: 8 examples - Less Wrong Discussion

48 Post author: gwern 18 May 2012 11:45PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (43)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 19 May 2012 10:05:55PM *  1 point [-]

This is something you can figure out from basic stats and your experimental design, and I strongly recommend actually running the numbers.

As it happens, I learned how to do basic power calculations not that long ago. I didn't do an explicit calculation for the melatonin trial because I didn't randomize selection, instead doing an alternating days design and not always following that, so I thought why bother doing one in retrospect?

But if we were to wave that away, the power seems fine. I have something like 141 days of data, of which around 90-100 is usable, giving me maybe <50 pairs? If I fire up R and load in the two means and the standard deviation (which I had left over from calculating the effect size), and then play with the numbers, then to get an 85% chance I could find an effect at p=0.01:

> pwr.t.test(d=(456.4783 - 407.5312) / 131.4656,power=0.85,sig.level=0.01,type="paired",alternative="greater")
Paired t test power calculation n = 84.3067
d = 0.3723187
sig.level = 0.01
power = 0.85
alternative = greater
NOTE: n is number of *pairs*

If I drop the p=0.01 for 0.05, it looks like I should have had a good shot at detecting the effect:

> pwr.t.test(d=(456.4783 - 407.5312) / 131.4656,power=0.85,sig.level=0.05,type="paired",alternative="greater")
Paired t test power calculation n = 53.24355

So, it's not great, but it's at least not terribly wrong?

EDIT: Just realized that I equivocated over days vs pairs in my existing power analyses; 1 was wrong, but I apparently avoided the error in another, phew.