Societies often punish people that refuse to help. Why not consider people that break the law as defectors?

In fact, that would be an alternative (and my preferred) way to fix you second and third objection to value ethics. Consider everyone who breaks the laws and norms within your community as a defector. Where I live, torture is illegal and most people think it's wrong to push the fat man, so pushing the fat man is (something like) breaking a norm.

Have you read Whose Utilitarianism?? Not sure if it addresses any of your concerns, but it's good and about utilitarianism.