It isn't crazy or mad to consider people who vote on your comments as on average equal to you in rationality. Quite the opposite: if each of us assumes that we are more rational than those who vote, this will be like everyone thinking that he is above average in driving ability or whatever.

And in fact, many people do use this information: numerous times someone has said something like, "Since my position is against community consensus I think I will have to modify it," or something along these lines.