Are you aware that downvotes are already limited by karma? Limiting upvotes as well might have merit.

There probably needs to be a bias towards upvotes however, otherwise it will be very difficult to get significant positive karma.