Our friends at Tricycle will push through a zero-based karma system (no self-voting possible) sometime this evening. At present this will only cover future posts/comments - they may go back and revise history some time in the indefinite future (or not), but apparently that would be overly complicated for now. We'll see how this works.
I agree with this, but with no limit on upvotes, the hypothetical spammer could just register more accounts to give each other lots of karma, then use that one to vote everything down. But then, that's a problem for this system too.
Are we just chasing ghosts?
For the moment, yes -- I don't see us being a high-enough priority target to be anticipating barbarians with torches and pitchforks. Still, if we go as far as we're hoping, it seems like a possibility.