gwern comments on Open thread, Oct. 6 - Oct. 12, 2014 - Less Wrong

6 Post author: MrMind 06 October 2014 08:16AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (332)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 10 October 2014 12:25:41AM *  7 points [-]

You could just answer it experimentally with A/B tests measuring time-on-page or leaving a review or something like that.

I see you're running off Apache on Dreamhost, so there's no doubt plenty of libraries to help you there, but there's other strategies: static sites can, with some effort, hook into Google Analytics to record the effect of an experiment, which is the approach I've used on gwern.net since I didn't want to manage a host like Dreamhost.

If I knew when I started what I know now, I would have begun with a large multifactorial experiment of the tweaks I tested one by one as I thought of them. It sounds like you are aware of the options you want to test, so you have it easy in that regard. (With the Abalytics approach, testing all those options simultaneously would be a major pain in the ass and probably hamper page loads since all possibilities need to be specified in advance in the HTML source, but I suspect any Apache library for A/B testing would make it much more painless to run many concurrent interventions.)

It'll probably take a few thousand visits before you have an idea of the larger effects, but that just means you need to leave the test running for a few months.