Just because you think an outcome is a thousand times more desirable, doesn't necessarily mean you would accept a 1,000:1 bet for it.
Correct. VNM utility is not necessarily linear with respect to the intuitive strength of the preference. Your utility function is defined based on what bets you would accept, rather than being a way of telling you what bets you should accept.
Like, 1,000 people dying seems like it should objectively be 1,000 worse than 1 person dying. Does that mean I should pay the mugger?
Nope; see above. You can define a notion of utility that corresponds to this kind of notion of importance, but this will not necessarily be the decision-theoretic notion of utility. For example, suppose an agent wants there to be many happy people, and thinks that the goodness of an outcome is proportional to the number of happy people, so it gives its utility function as U(there are n happy people) = n. And suppose it has the following way of assigning utilities to uncertain outcomes: It picks some strictly increasing continuous function f (which could be arctan, for instance), it calculates the expected value of f(n), and applies f^-1 to that to get the utility. Assuming f is nonlinear, this agent does not use the mean utility as the utility of a gamble, but it is still VNM-rational, and thus by the VNM theorem, there exists a utility function V (not the same as U), such that the agent acts as if it was maximizing the expected value of V; this utility function is given by V(there are n happy people) = f(n).
In a previous post, I looked at some of the properties of using the median rather than the mean.
Inspired by Househalter's comment, it seems we might be able to take a compromise between median and mean. It seems to me that simply taking the mean of the lower quartile, median, and upper quartile would also have the nice features I described, and would likely be closer to the mean.
Furthermore, there's no reason to stop there. We can take the mean of the n-1 n-quantiles.
Two questions:
Note the unlike the median approach, for large enough n, this maximiser will pay Pascal's mugger.