I, too, sometimes read from a user's user-page and up- or down-vote comments according to my impressions of their individual quality.

This wouldn't create the -50 karma jump conchis describes, at least the way I do it, but it is evidence that orthonormal's suggestion would have some costs.