Downvotes imply net disapproval, particularly for someone whose comments get read as much as Eliezer's. If you think of it as simply losing points, it seems trivial, but if you take it as a sign that "people seem not to like it when I do that," it's a meaningful consideration.