My point was really that karma isn't tied to who is right, it is tied to who we like or who furthers our preferential ends.

Barring an objective method for telling what arguments are right, this is the way any human-run evaluation system (including e.g. formal peer review or university grades) has to work. You can try to eliminate the "who we like" part by trying to blind the identities of the people in question, but since one cannot assess degree-of-correctness directly, one has to rely on some other criteria, e.g. the extent to which the comment seems to take previous work into account. And those other criteria and their parameters, like what counts as "previous work", are ultimately set by an in-group consensus. (I felt that James Paul Gee had a particularly good elaboration of this.)

But I do agree that awarding karma for work distances karma from correctness even further than would be necessary. (Not sure whether it's a bad thing, though.)