This could just make implementation of a polling system a dependence. Total Karma probably wouldn't work, most of the edge cases where it's not totally obvious that the user should be removed allow for a positive average balance. Also, one of the use cases applies to established contributors going into a wrong mode, in which case they'd be quite capable of offsetting the downvotes.