= 27d567bc2d48d9f40e9ccc20ea370b15)
Douglas_Knight comments on Open Thread, June 16-30, 2012 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (344)
After a painful evening, I got an A/B test going on my site using Google Website Optimizer*: testing the CSS
max-widthproperty (800, 900, 1000, 1200, 1300, & 1400px). I noticed that most sites seem to set it much more narrowly than I did, eg. Readability. I set the 'conversion' target to be a 40-second timeout, as a way of measuring 'are you still reading this?'Overnight each variation got ~60 visitors. The original 1400px converts at 67.2% ± 11% while the top candidate 1300px converts at 82.3% ± 9.0% (an improvement of 22.4%) with an estimated 92.9% chance of beating the original. This suggests that a switch would materially increase how much time people spend reading my stuff.
(The other widths: currently, 1000px: 71.0% ± 10%; 900px: 68.1% ± 10%; 1200px: 66.7% ± 11%; 800px: 64.2% ± 11%.)
This is pretty cool - I was blind but now can see - yet I can't help but wonder about the limits. Has anyone else thoroughly A/B-tested their personal sites? At what point do diminishing returns set in?
* I would prefer to use Optimizely or Visual Website Optimizer, but they charge just ludicrous sums: if I wanted to test my 50k monthly visitors, I'd be paying hundreds of dollars a month!
Do you know the size of your readers' windows?
How is the 93% calculated? Does it correct for multiple comparisons?
Given some outside knowledge, that these 6 choices are not unrelated, but come from a ordered space of choices, the result that one value is special and all the others produce identical results is implausible. I predict that it is a fluke.
Yes, I'm finding the result odd. I really did expect some sort of inverted V result where a medium sized max-width was "just right". Unfortunately, with a doubling of the sample size, the ordering remains pretty much the same: 1300px beats everyone, with 900px passing 1200px and 1100px. I'm starting to wonder if maybe there's 2 distinct populations of users - maybe desktop users with wide screens and then smartphones? Doesn't quite make sense since the phones should be setting their own width but...
A bimodal distribution wouldn't surprise me. What I don't believe is a spike in the middle of a plain. If you had chosen increments of 200, the 1300 spike would have been completely invisible!