Why do we feel that a downvote is such a big deal? A downvote doesn't mean "we think your an idiot, go jump off a bridge or at least don't ever post here again". It means that this comment isn't up to our standards, and you should do better next time.

It's a big deal if it's the first or only vote on a comment, because it puts the comment into the negative. A negative score by itself presents the message, "the community considers your contribution a net negative".

I suspect, though, that if the same score were displayed as "+0/-1", it would communicate a VERY different message, i.e., "One person didn't like this, and nobody else cared one way or another."

Those two messages are likely to produce different effects in most people's brains.