More than five times ever? That's going to be a problem for long-term participants.

More than five times in some fixed period? That would make more sense, though it might need extra information recorded in the database. The right way might be something like this. For each pair of users, store an "amount of recent downvoting" that decays exponentially over time and gets incremented with each downvote; then apply a karma toll that increases with the value of this. It would also be necessary to store the last time the downvote density was updated, so that the exponential decay can be applied properly.