My experience is similar. On especially controversial or social-political subjects I've sometimes found an early downvote to be a predictor of a higher later karma score, a far cry from a negative spiral.