One way to encourage people to not just vote stuff up because it is already popular would be to occasionally put up a subtle nonsense post, preferably by someone high status like EY or robin, seed it with an already fairly high score (such as 10) and then heavily penalize people's karma if they vote it up. One might call this a "honeypot for phoney rationalists"
This would require some incentive for people to vote, to compensate for the small probability of being hit with a penalty. Overall this would make the Karma system more complex.
Should we be worried that people will vote stuff up just because it is already popular? There is currently no penalty for voting against the crowd, so wouldn't people (rightly) want to do this?
(Of course, we assume people are voting based on their personal impressions. It's clear that votes bases on Bayesian beliefs are not are useful here.)
An information cascade is a problem in group rationality. Wikipedia has excellent introductions and links about the phenomenon, but here is a meta-ish example using likelihood ratios.
Suppose in some future version of this site, there are several well-known facts:
Let's talk about how the very first reader would vote. If they judged the post high quality, then they would multiply the prior likelihood ratio (6:4) times the bayes factor for a high private signal (4:1), get (6*4:4*1) = (6:1) and vote the post up. If they judged the post low quality then they would instead multiply by the bayes factor for a low private signal (1:4), get (6*1:4*4) = (3:8) and vote the post down.
There were two scenarios for the first reader (private information high or low). If we speculate that the first reader did in fact vote up, then there are two scenarios for the second scenario: There are two scenarios for the second reader:
Note that now there are two explanations for ending up two votes up. It could be that the second reader actually agreed, or it could be that the second reader was following the first reader and the prior against their personal judgement. That means that the third reader gets zero information from the second reader's personal judgement! The two scenarios for the third reader, and every future reader, are exactly analogous to the two scenarios for the second reader.
This has been a nightmare scenario of groupthink afflicting even diligent bayesians. Possible conclusions:
Note: Olle found an error that necessitated a rewrite. I apologize.