People with 0 Karma can still upvote, so there is a danger that new members will just upvote each other quickly. This problem could be solved if there is enough latency between a point where a user starts participating and where they can influence moderation by voting, i.e. only allow voting (both upvoting and downvoting) after a significant threshold, something like 500 Karma points, so that it would take at least a month at typical Karma gain rates of very active users to get there, sufficient for vetting of new members.

I made a very similar proposal recently, just set at 1000 karma. There is some discussion there on the ups and downs of such an approach behind the link.