As an alternative to you just making the decision to ban users in order to improve site's quality, maybe establish a user-banning board? Known users on the board, with information about individual voting not publicly available. These cases are sufficiently rare and different for development of an adequate automated system being very difficult to impossible.

(Another alternative is to open such decisions to an open vote and discussion, but this can be noisy and with other undesirable consequences, probably even worse than authoritarian system.)