I think that impact of an upvote or a downvote should be inversely proportional to how often that person votes.

That would be simple to implement in the form of giving each user, say, 10 votes per day, non-accumulating.