I don't think that I use poster-karma much to gauge how useful a comment is likely to be -- I do have a rough mental ordering of how competent I think people are in different areas, but this is based on reading their writing, not their karma. So unless you're trying to lower their karma to the extent that they have trouble posting, reducing people's karma doesn't have much of an impression on me.

OTOH, I do use the karma levels of posts to rank their likely usefulness -- I usually sort the comments of a post by 'Top' and only read a portion of the comments (especially for posts with lots of comments.) As such, downvoting comments because of who posted them instead of for their content is harmful to me. To anyone who is doing it, please stop.

tl;dr: Agree.