wedrifid comments on Better to be testably wrong than to generate nontestable wrongness - Less Wrong

-6 Post author: Dmytry 20 March 2012 07:04PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (23)

You are viewing a single comment's thread. Show more comments above.

Comment author: satt 21 March 2012 04:17:16AM 1 point [-]

I'm actually finding this hypothesis more interesting than the one in your OP (partly because it looks more testable, funnily enough). Bash out a script to watch LW and vote on things as they appear, leave it to generate data as long as one likes, then hey presto. Tiny bit tempted to do it myself, approval or not.

The test is conducted in a random week (edit: not sure how many data points you can get out of a week worth of comments tho, may require longer or shorter time).

The sample size you need to detect an effect depends on that effect's size. So far, so obvious, so I did a quick & dirty power analysis to get some numbers, although for posts in the discussion section rather than comments. (Posts on main are too infrequent, and I'd expect a smaller effect for comments, so comments would need a bigger sample.) If anyone cares I can throw up my code.

If my numbers are right and you took a sample of 100 upvoted posts and 100 downvoted discussion posts, the bootstrap confidence interval for the effect size would be 3.7-6.8 points wide. Even with a sample of 400 upvoted posts and 400 downvoted (and that's 3-4 months' worth of discussion posts), it'd be 2.2-3.0 points wide. So unless the priming effect's strong (at least 2-4 points) a week of data wouldn't be conclusive, at least not for posts. Comments might be more doable, though.

Comment author: wedrifid 21 March 2012 04:19:42AM 0 points [-]

Bash out a script to watch LW and vote on things as they appear, leave it to generate data as long as one likes, then hey presto. Tiny bit tempted to do it myself, approval or not.

I'm becoming increasingly tempted to submit automation detection scripts to the lesswrong codebase.