He didn't mean your karma specifically, but Arbitrary Hypothetical Second-Person Comment-Poster X's karma.

For example, suppose E.Y. were to post, for whatever reason (cat jumping on keyboard?), a really pointless, flawed comment that everyone actually downvoted. If the algorithm takes into account the poster's total Karma in some proportional manner not implementing any diminutive return strategy, then E.Y.'s downvoted comment would still be at the top unless it received an amount of downvotes that seems almost impossible to obtain using only the current active LW readership.