Well, ideally if a moderator deleted your comment as spam you'd get your karma back; or maybe downvoting shouldn't be used as a way to deal with spam (there is a "report" button for that).