If post a response to someone, and someone replies to me, and they get a single silent downvote prior to me reading their response, I find myself reflexively upvoting them just so they won't think I was the one who did the single silent downvote, since it seems plausible to me that if you have a single downvote, and no responses, the most likely explanation to me was that it was from the person who you replied to downvoted you, and I don't want people to think that.

Except, then I seem to have gotten my opinion of the post hopelessly biased before even reading it, because I'd feel bad if I revoked the upvote, let alone actually downvoted them, and I feel like I can't get back to the status quo of them just having a 0 point or positive post.

It also doesn't seem like it would have the same effect if someone replied to me and was heavily downvoted, but I don't actually recall that happening.

If I try to assess this more rationally, I get the suggestion 'You're worrying far too much about what other people MIGHT be thinking, based on flimsy evidence."

Thoughts?