satt comments on Stuff That Makes Stuff Happen - Less Wrong

51 Post author: Eliezer_Yudkowsky 18 October 2012 10:49AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (127)
Tags: antitheism causality seq_epistemology anti_epistemology

Comments (127)

You are viewing a single comment's thread.

Comment author: satt 17 October 2012 11:39:55PM 14 points [-]

It's worth noting that the Center for Applied Rationality ran the June minicamp experiment using a standard but unusual statistical method of sorting applicants into pairs that seemed of roughly matched prior ability / prior expected outcome, and then flipping a coin to pick one member of each pair to be admitted or not

As an aside, if you're interested in looking up more about this nifty experimental design trick, the magic keyword is "blocking". The idea of randomized block designs dates back to Fisher.

Comment author: gwern 18 October 2012 05:53:06PM 6 points [-]

I've found blocking to be really useful for my small-scale experiments for 2 different reasons:

  1. Often, I'm worried about simple randomization leading to an imbalance in sample vs experimental; if I'm only getting 20 total datapoints on something, then randomization could easily lead to something like 14 control and 6 experimental datapoints - throwing out a lot of statistical power compared to 10 control and 10 experimental. If I pair days, then I know I will get 10/10, without worrying about breaking blinding.
  2. Blocking is the natural way to handle multiple-day effects or trends: if I think lithium operates slowly, I will pair entire weeks or months, rather than days and hoping enough experimental and control days form runs which will reveal any trend rather than wash it out in averaging.
Powered by Reddit Powered by Reddit