I don't use karma to keep score, only as one of many sources of information about other people's opinion. When one of the comments I've posted gets a much different response than I expected, it is my expectation that was at fault.

Note that one reason to be wrong about how people will respond to a given stimulus is to overestimate how much of a desirable trait that they have. So while "your expectation is at fault" to the extent that correctly predicting response is a goal, the update in such circumstance is still in the direction of less respect for the people you are attempting to predict. That can be disheartening at times.