Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

# New "Best" comment sorting system

25 02 July 2012 11:08AM

Way back in October 2009 Reddit introduced their "Best" comment sorting system. We've just pulled those changes into Less Wrong. The changes affect only comments, not stories.

It's good. It should significantly improve the visibility of good comments posted later in the life of an article. You (yes you) should adopt it. It's the default for new users.

See http://blog.reddit.com/2009/10/reddits-new-comment-sorting-system.html for the details.

Sort By: Best
Comment author: 02 July 2012 06:40:47PM *  13 points [-]

Short version of how this is different, for those too lazy to click on the link: if you sort by "top", comments get sorted in a simple "the ones with the highest score go on top" order. This has the problem that it favors comments that were posted early on, since they're the ones that people see first and they've had a lot of time to gather upvotes. A good comment that's posted late might get stuck near the bottom because few people ever scroll all the way down to upvote it.

"Best" uses some statistical magic to fix that:

If everyone got a chance to see a comment and vote on it, it would get some proportion of upvotes to downvotes. This algorithm treats the vote count as a statistical sampling of a hypothetical full vote by everyone, much as in an opinion poll. It uses this to calculate the 95% confidence score for the comment. That is, it gives the comment a provisional ranking that it is 95% sure it will get to. The more votes, the closer the 95% confidence score gets to the actual score.

Not sure I fully understood that either. But they say it works well, so I guess I'll trust them!

Comment author: 02 July 2012 08:03:19PM *  6 points [-]

"Best" uses some statistical magic to fix that:

I'm curious whether the math still works correctly on a site where the default karma is 1 instead of 0. But since it's magic to start with, I guess "meh". Let's just not use it to calculate CEV or anything. ;-)

Comment author: 03 July 2012 02:28:42AM *  3 points [-]

I think what they're doing is doing statistical inference for the fraction upvotes/total_votes. I'm not sure this is the best model, possible but it seems to have worked well enough.

I suspect they're taking the mean of the 95% confidence interval, but I'm not sure. There's actually a pretty natural way to do this more rigorously in a Bayesian framework, called hierarchical modeling (similar to this), but it can be complex to fit such a model.

Edit: However, a simpler Bayesian approach would just be to do inference for a proportion using a 'reasonable' prior for the proportion (which approximates the actual distribution of proportions) expressed as a Beta distribution (this makes the math easy). Come to think of it, this would actually be pretty easy to implement. You could even fit a full hierarchical model using a data set and then use the prior for the proportion you get from that in your algorithm. The advantage to this is that you can do the full hierarchical model offline in R and avoid having to do expensive tasks repeatedly and having to code up the fitting code. The rest of the math is very simple. This idea is simple enough that I bet someone else has done it.

Comment author: 03 July 2012 06:45:17AM *  5 points [-]

If you use the Bayes approach with a Beta(x,y) prior, all you do is for each post add x to the # of upvotes, add y to the # of downvotes, and then compute the % of votes which are upvotes. [1]

In my college AI class we used this exact method with x=y=1 to adjust for low sample size. Someone should switch out the clunky frequentist method reddit apparently uses with this Bayesian method!

[1] This seems to be what it says in the pdf.

Comment author: 02 July 2012 04:46:14PM 5 points [-]

How do the "Best", "Popular", and "Top" algorithms work?

Comment author: 02 July 2012 06:22:22PM 5 points [-]

Ironically, it appears the new algorithm is frequentist.

Comment author: 02 July 2012 11:00:39PM 3 points [-]

Bayesian reformulations welcome.

Comment author: 04 July 2012 08:07:08AM 1 point [-]

Apologies â€” I should have taken reinforcement into account and noted that the new algorithm is probably still a lot better than the previous one.

Comment author: 03 July 2012 03:02:14AM 0 points [-]

This seems like a neat problem. Would it be hard to go from a python function that takes a set of comment upvote downvote counts and returns a ranking to a comment sorting option? If I don't know much about the reddit internals?

Also, would it be difficult to get a real dataset of comment counts from LW?

Comment author: 02 July 2012 06:46:56PM *  2 points [-]

"Top" simply calculates the (number of upvotes - number of downvotes) and puts on top the comments that rank the highest this way.

I think "Popular" tries to favor comments that don't have many downvotes or something, I'm not sure.

"Best" apparently works by magic.

Comment author: 02 July 2012 10:56:57PM *  2 points [-]

I think "Popular" adds weight to recent comments. This seems to be a much worse way of achieving what "Best" shoots for.

Comment author: 03 November 2014 02:00:38AM *  0 points [-]

This seems to be a much worse way of achieving what "Best" shoots for.

Not necessarily. Someone who has already seen the best comments and returns a while later to see what new but good comments have been posted may have a use for it.

Comment author: 02 July 2012 11:59:33AM 3 points [-]

Yay! Thanks Matt and the tricycle team (and anyone else) for continuing to improve LW.

Comment author: 02 July 2012 10:59:52PM 2 points [-]

Work done by John Simon, and integrated by Wes.

Comment author: 03 July 2012 07:25:08AM 0 points [-]

Thanks, John and Wes!

Comment author: 02 July 2012 12:50:52PM 1 point [-]

Thanks Matt!

Comment author: 02 July 2012 12:03:32PM 1 point [-]

Thank you for taking the time to implement this, I've set it as my "sort by" criteria.

Comment author: 22 November 2014 07:33:16PM *  0 points [-]

For me, this is not working for some of the posts,

Comment author: 03 November 2014 02:42:15AM 0 points [-]

I found a Reddit thread explaining the different comment sorting systems. Does LW use the same algorithms for each method?