The relative utilities are staggering: I wouldn't allow a mob of 100,000 to kill another human no matter how much they wanted to and even if their quality of life was improved (up to a point).

The trouble is that pretty much all decisions that get punted far enough upstairs that they are handled by people whose job description is "politician" are trolley problems - lose-lose hypotheticals. More people die or live worse versus less people dying or living worse - which people, where, when, to what degree?

So your refusal to countenance such doesn't stop such decisions existing - it just means you won't be the one personally deciding. But as long as you benefit from the decision (e.g. live in society), I don't think this actually makes you a morally purer person.