Why are downvotes bad? Because:

I agree with enough of this. I know there are immediate downsides and hypothetical dangers. But the upsides seem indispensable. The argument needs to consider the balance of the two.

If you don't want to talk to someone, don't upvote her and don't reply to her.

They remain in the fabric of the forum, making it less fun to read. Not upvoting doesn't address this issue.

"Safe spaces" are about banning content which might offend someone's sensibilities. My suggestion is about "banning" less content.

Things that are not fun (for certain sense of "fun") offend my sensibilities (for certain sense of "offend"). My suggestion is to discourage them by downvoting. (This is the intended analogy, which is strong enough to carry over a lot of Yvain's discussion, even if the concept "safe spaces" doesn't apply in detail, although I think it does to a greater extent than I think you think it does.)