I would actually kind of like it if it did cost karma to downvote. It would mean something to me when I downvoted a post that was irrational or in my judgment out of line socially.

What would be even better is if it were possible to downvote comments multiple times at an exponentially increasing cost. There are some comments that are in such contrast to my preferences that I would be willing to spend 63 votes (1+2+4+8+16+32) to down-vote 6 times. This would sometimes reduce the need to do 'costly signaling' in the form of comments with strongly assertive and direct criticisms.

I can imagine myself thinking "No, that is blatant bullshit but I know if I call you on it you will throw more bullshit and drag me in to your drama. I am willing to pay 31 karma to discourage the bullshit without the (minor) physiological and psychological stress of engaging in social conflict with unreasonable people."