You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

gwern comments on Open thread, Sep. 19 - Sep. 25, 2016 - Less Wrong Discussion

2 Post author: DataPacRat 19 September 2016 06:34PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (92)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 24 September 2016 12:50:48AM 0 points [-]

I think that it is clear that can't do it just splitting existing control group in two parts, as such action could be done in many different ways and researcher could choose most favorable, and also because there could be some interactions inside control group, and also because smaler statistic power.

You can. Cross-validation, the bootstrap, permutation tests - these rely on that sort of procedure. They generate an empirical distribution of differences between groups or effect sizes which replace the assumption of being two normal distributions etc. It would be better to do those with both the experimental and control data, though.