It seems to me that the total voting ought to reflect the "net reward" we want to give the poster for their action of posting, like a trainer rewarding good behavior or punishing bad. For this reason, my voting usually takes into account the current total score. I think the community already abides by this for most negatively scored posts -- they usually don't sail much below -2. For posts that I feel I really benefited from, though, I don't really follow my own policy per se. -- I just "pay back" what I got out of it to them.

Where do I set the threshold of good/bad?

I basically only downvote if there's some line of argument that I object to in the post. I think I need to say what I'm objecting to specifically when I do this more often.

Is it best for the community if I upvote only exceptionally good comments, or downvote only very bad comments, or downvote all comments that aren't exceptionally good, or something else?

My opinion is it has to depend on the current score of the post. [At least under the current system, which reports, if you will, net organic responses; in a different system where responses were from solicited peer-review requests, different behavior would be warranted.]

Has this been studied? Is it possible to make a karma system where this question doesn't arise?

Good questions. I don't know. There's some further discussion here.