For what it's worth, I haven't noticed that myself, and I don't think it's ever happened to me here. But I agree that when it happens, it's an example of the problem you're talking about.

Downvoting also allows people to express disagreement without having to give reasons or even pay much attention to what's said. I think this also goes against the purpose of the site.

I agree with this too. I think maybe we have just different intuitions of how commonly it's actually used like that.