LESSWRONG
LW

Comment Permalink
Taran40

It gives a reasonably rigorous way of predicting how many upvotes and downvotes a post will get, given the history of the user who wrote it. Specifically, it defines a probabilistic model: for each user, we can specify a Beta distribution with various unknown parameters, and then learn those parameters from the user's post history. The details of that learning are rather charming if you're a statistician, or aspire to be one, but don't translate very well.

mr-hire would like to know what his particular Beta distribution looks like. To find out, we have to adapt Moulton's method to the LW karma system. This turns out to be a little difficult, and requires some additional modeling choices:

Moulton models votes on individual posts with a Binomial distribution, which is used for sequences of binary outcomes. In this case each voter either upvotes the post (with probability p) or downvotes it (with probability 1-p) -- we ignore non-voters since it's hard to know how many of them there are. But a LessWrong voter has four choices: they can vote Up or Down, and they can vote Normal or Strong, so the Binomial distribution is no longer appropriate.

This is fixable with a different choice of distributions, but then you run into another problem. In LW, even normal votes vary in value: an upvote from a high-karma user is worth more than one from a low-karma user. Do we wish to model this effect, and if so how?

If you were willing to treat all user votes equally I think you could get away with using the Dirichlet-multinomial. If not, I think you have to give up on modeling individual votes and try to model karma directly, without breaking it down into its component upvotes and downvotes.

Reply
See in context
by jmh
1 min read
A
114

4

I was thinking the other day it would be interesting to know something about the karma I was getting. Particularly the quality, to the extent that is possible to assess.

I suspect everyone would prefer to have those they respect up voting posts and comments but that type of transparency might not be desirable. I'm not sure if some type of weighting system would work or not -- how may karma points do the account giving an up vote have so you can get some average gauge of quality of the feedback.

I would not think that should be on the main page but something each person can look at under their own account.

Wonder what others think or if this has been suggested before?

Site Meta
Personal Blog

4

Source of Karma
New Answer
New Comment

1 Answers sorted by
top scoring

Dagon

30

I'd much rather have less focus on meaningless internet points around here (and in most places). Focus on collecting good comments that help you update your beliefs and models, in order to be less wrong.

Note that there _is_ a weighting that happens - higher-karma people give/take more than 1 karma with their votes. It's not specific to your evaluation of them, nor all that visible, but it's there.

[-]jmh10

I think there are two aspects to this though. One the comments that any person finds useful in making better decisions later and evaluation of your own comments -- sometimes we get points without any comment.

It was this last aspect I was interested in. If I get no feedback other than the vote I have little useful information. It's also reasonable,then, for me to simply ignore it, but in that case why bother with having it here? (I should check the setting to see if I can just turn the display off perhaps).

I suppose I can also check to see if the vote... (read more)

Reply
6Viliam
That would increase the voting time by orders of magnitude, which would result in fewer votes. The fewer votes there are, the more random the outcome. Making the votes non-anonymous (which is what a mandatory vote comment would mean) would open yet another layer of karma obsession. People replying to vote comments, people complaining about voting patterns of other people, people taking revenge for downvotes or forming mutually upvoting cliques (not necessarily consciously).
1jmh
This is a rather disheartening answer. It might be completely true but... If the community here is so petty then I suspect the experiment to both improve one's on decision making and help others do the same has probably failed. I don't really think that is the case though. If my view is correct then even with fewer votes they will be higher quality votes that everyone will take seriously and not spin off into a bunch of factional fighting -- or at least no more than already exists. But I think for me the answer is just turn the display off in the header line and to the extent I can just ignore the votes on anything as it appears to be of such little value.
8Raemon
The thing I think it is most appropriate to think of votes as meaning is something like "people in a conversation smiling at me, or kinda leaning away from me, or looking a bit annoyed,  or maybe calling their friend over to join the conversation." Having karma gives you a rough sense of how people are responding to you. This isn't super detailed information, but I do think it's better than not having it. If you asked people not to smile, or frown, without giving an explanation why, they'd end up adopting neutral expressions all the time. I don't think it's at all 'petty' that people have limited time and would give less feedback. (Unless you were specifically referring to the 'people complaining about voting patterns and taking revenge for downvotes part'. In which case I agree that would reflect poorly, but also I'm skeptical about claims that this is so important that it means the project has failed.) I do currently lean towards requiring reasons for Strong Votes, but the whole point of regular votes is to be less effort than commenting. I might be open to a more general version where providing a reason for your vote gives the vote a bit more weight, or something, but that's more work to implement.
1jmh
That is a good view I think. I think perhaps just understanding how the voters karma really impacts their vote would be helpful. I was unaware of that until Dagon mentioned it but it's a black box to me and I have no clue of what the output of the box is. My original thought was prompted by the idea we might get more value if we could assess the quality of the vote. If I up or down vote anything I don't think that should be a valuable indicator; I am not the bight bulb in the room. However, there are a lot of bight bulbs here so thinking their vote would carry the same weight as mine seems lacking -- just from a purely selfish self assessment of reactions to my input. Can anyone share what the weighting on votes is?
3Raemon
This blogpost, which explains the voting weights, might be what you're looking for.
1jmh
Thanks! Interesting read and helpful to understand. You get an up vote ;-)
6Viliam
Look at the karma numbers in this debate, and imagine them divided by ten. Oops, nothing is left. Now imagine the same thing, except that one person (for whatever reason) bothered to vote. Now that one person's opinion is all the feedback you have. Also, that person is probably going to be someone with too much free time. (For the record, I agree that better feedback would be nice to have, it's just that I find this cure worse than the original problem. The problem is that better feedback is costly in terms of time and effort, and when you increase the costs, instead of better feedback you simply get less feedback. I mean, currently nothing is preventing the people who vote from also writing a comment. I am also pessimistic about finding a simple solution that would improve things, mostly because I think that if such simple fix existed, someone else would have already tried it on a different website.)
2Dagon
Yup, voting systems with no mechanism attached (nothing to do with the votes, no outcome decided by them) have very little information, for very little cost. It's unclear what, if any, changes will significantly increase the information without significantly increasing the cost.
4 comments, sorted by
top scoring
Click to highlight new comments since:
[-]Matt Goldenberg30

Recently ran across this post that recommends using a Beta-Binomial distribution to more correctly represent the uncertainty that any given post adds to your overall karma. I thought it was a cool idea and would love to see what my Karma is when represented that way, rather than just adding everything together:

https://moultano.wordpress.com/2013/08/21/how-karma-should-work-betabinomial/

Reply
[-]Raemon20

I'd be interested in a mathy person translating that into less mathy language that roughly communicates why it's interesting. 

Reply
[-]Taran40

It gives a reasonably rigorous way of predicting how many upvotes and downvotes a post will get, given the history of the user who wrote it. Specifically, it defines a probabilistic model: for each user, we can specify a Beta distribution with various unknown parameters, and then learn those parameters from the user's post history. The details of that learning are rather charming if you're a statistician, or aspire to be one, but don't translate very well.

mr-hire would like to know what his particular Beta distribution looks like. To find out, we have to adapt Moulton's method to the LW karma system. This turns out to be a little difficult, and requires some additional modeling choices:

Moulton models votes on individual posts with a Binomial distribution, which is used for sequences of binary outcomes. In this case each voter either upvotes the post (with probability p) or downvotes it (with probability 1-p) -- we ignore non-voters since it's hard to know how many of them there are. But a LessWrong voter has four choices: they can vote Up or Down, and they can vote Normal or Strong, so the Binomial distribution is no longer appropriate.

This is fixable with a different choice of distributions, but then you run into another problem. In LW, even normal votes vary in value: an upvote from a high-karma user is worth more than one from a low-karma user. Do we wish to model this effect, and if so how?

If you were willing to treat all user votes equally I think you could get away with using the Dirichlet-multinomial. If not, I think you have to give up on modeling individual votes and try to model karma directly, without breaking it down into its component upvotes and downvotes.

Reply
[-]jmh10

For me, just understanding the weighting for the votes based on existing karma of the voter would be sufficient to say my idea was "implemented". Even without knowing that it was helpful to learn about that feature of the votes.

Reply
Moderation Log
More from jmh
View more
Curated and popular this week
A
14