Psst. If this comment's parent and its grandparent have different absolute karma counts, someone is doing the poll wrong -- the karma is supposed to balance out.

Karma balancers are there to give people the option of contributing to the poll without the obligation of giving karma to the poll provider. Not downvoting the balancer is the obvious way to vote up the poll itself in the sense that one would normally provide karma to a comment - approval, disapproval, reward, punishment, etc. Downvoting the balancer without upvoting the 'vote up' part is how one would express disapproval of having the poll at all, without sabotaging the results.