gwern comments on Stuff That Makes Stuff Happen - Less Wrong

51 Post author: Eliezer_Yudkowsky 18 October 2012 10:49AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (127)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 18 October 2012 05:53:06PM 6 points [-]

I've found blocking to be really useful for my small-scale experiments for 2 different reasons:

  1. Often, I'm worried about simple randomization leading to an imbalance in sample vs experimental; if I'm only getting 20 total datapoints on something, then randomization could easily lead to something like 14 control and 6 experimental datapoints - throwing out a lot of statistical power compared to 10 control and 10 experimental. If I pair days, then I know I will get 10/10, without worrying about breaking blinding.
  2. Blocking is the natural way to handle multiple-day effects or trends: if I think lithium operates slowly, I will pair entire weeks or months, rather than days and hoping enough experimental and control days form runs which will reveal any trend rather than wash it out in averaging.