I was too lazy and tired to parse your sentence and replied to the argument I would have liked to be refuted.

If I understand the point in question it seems we are in agreement - voting is evidence about the reasoning of the voter which can in turn be evidence about the comment itself. In the case of downvotes (and this is where we disagree), I actually think it is better that we don't have access to that evidence. Mostly because down that road lies politics and partly because people don't all have the same criteria for voting. There is a difference between "I think the comment should be at +4 but it is currently at +6", "I think this comment contains bad reasoning", "this comment is on the opposing side of the argument", "this comment is of lesser quality than the parent and/or child" and "I am reciprocating voting behavior". Down this road lies madness.