Guilty as charged.

Showing upvotes and downvotes separately would stop me (and, probably, other people) from doing that, but I'm kind of getting tired of beating this dead horse.

True. This is a problem since the current net vote count is mutable, while an individual vote, once cast, is not. You could try fitting a much more complicated model that can reproduce this behavior, calibrate it with A/B testing, etc. Or maybe try to prevent it by sorting according to quality, but not actually displaying the metrics.