I don't think a single downvote is that significant. I've seen so many inexplicable downvotes that I would suspect there's a bot that every time a new comment is posted generates a random number between 0 and 1 and downvotes the comment if the random number is less than 0.05 -- if I could see any reason why anyone would do that.

It actually depends on how many total votes the post in question has accumulated, and how much karma the user in question has accumulated.

A completely new user doesn't need to have to worry about anomalous downvotes, because they're too new to have a reputation. Likewise, a well-established user doesn't need to worry about anomalous downvotes, because they get lost in the underflow. I'd say somewhere around the 200 mark it can become problematic to one's reputation; and, of course, one or two consecutive anomalous downvotes that act as the first votes on a given post can easily set a trend to bury that post before anyone has a chance to usefully comment on it.