I think that asking why a comment was downvoted would be legitimate even more than two times in a row, were the downvoted comments downvoted more than once.

For comments that were only downvoted once, it is not usually a question worth asking. So I agree with the literal reading of the original "don't ask this for every comment of yours that is downvoted" more than the clarification.