You raise a good point. But then who will judge when evidence should be downvoted or not? There is probably no easy answer to that. My point is: the voting system is flawed and maybe we should rethink it or at least put certain safeguards in place.
But then who will judge when evidence should be downvoted or not?
How about the people doing the downvotes?
People who go back and downvote every post or comment a Less Wrong user has ever made, please, stop doing that. It's a clever way to pull information cascades in your direction but it is clearly an abuse of the content filtering system. It's also highly dishonorable. If you truly must use such tactics then downvoting a few of your enemy's top level posts is much less evil; your enemy loses the karma and takes the hint without your severely biasing the public perception of Less Wrong's discourse.
(I just lost over 200 karma in a few minutes and that'll probably continue for awhile. This happens to me every few weeks. Edit: I mean it's been happening every few weeks for a few months for a total of only three or four. Between 400 and 700 karma lost total I think? I don't mean to overstate the problem.)