if people downvote what they don't understand, and it's a good comment, then it should have more upvotes than downvotes if most people understand it. If it has more downvotes than upvotes in this scenario, then it was not explained well enough for the majority of readers.

These are generalizations, of course, and depend largely on actual voting habits. But so was the note that it will pull the discussion to the level of the 'least competent people' - possibly the same observation could be stated as pulling the discussion to the level of the majority of the readership.