It may well do. Yvain has pointed out on his blog (I recall the post, though I couldn't find it just now) that in daily life we do actually use something like utilitarianism quite a bit, which carries a presumption of something like a utility function at least in that case. But what works in normal ranges does not necessarily extrapolate: utilitarianism is observably brittle, and routinely reaches conclusions that humans consider absurd.
There's occasionally LW posts showing that utilitarianism gives some apparently-absurd result or other, and too often the poster seems to be saying "look, absurd result, but the numbers work out so this is important!" rather than "oh, I hit an absurdity, perhaps I'm stretching this way further than it goes." It's entirely unclear to me that pretending you're an agent with a utility function is actually a good idea; it seems to me to be setting yourself up to fall into absurdities.
Below, you claim this is a moral choice; I would suggest that trying to achieve an actually impossible moral code, let alone advocating it, is basically unhealthy.
...too often the poster seems to be saying "look, absurd result, but the numbers work out so this is important!" rather than "oh, I hit an absurdity, perhaps I'm stretching this way further than it goes."
Yes, I don't understand this at all. For example, even Yudkowsky writes that he would sooner question his grasp of "rationality" than give five dollars to a Pascal's Mugger because he thought it was "rational". Now as far as I can tell, they still use this framework to make decisions, a framework that implies absu...
I think we should stop talking about utility functions.
In the context of ethics for humans, anyway. In practice I find utility functions to be, at best, an occasionally useful metaphor for discussions about ethics but, at worst, an idea that some people start taking too seriously and which actively makes them worse at reasoning about ethics. To the extent that we care about causing people to become better at reasoning about ethics, it seems like we ought to be able to do better than this.
The funny part is that the failure mode I worry the most about is already an entrenched part of the Sequences: it's fake utility functions. The soft failure is people who think they know what their utility function is and say bizarre things about what this implies that they, or perhaps all people, ought to do. The hard failure is people who think they know what their utility function is and then do bizarre things. I hope the hard failure is not very common.
It seems worth reflecting on the fact that the point of the foundational LW material discussing utility functions was to make people better at reasoning about AI behavior and not about human behavior.