I like this idea, but I would also, it seems, need to consider the (probabilistic) length of time each utility function would last.

That doesn't change your basic point, though, which seems reasonable.

The one question I have is this: In cases where I can choose whether or not to change my utility function - cases where I can choose to an extent the probability of a configuration appearing - couldn't I maximize expected utility by arranging for my most-likely utility function at any given time to match the most-likely universe at that time? It seems that would make life utterly pointless, but I don't have a rational basis for that - it's just a reflexive emotional response to the suggestion.

So it seems to me that the solution is use an expected utility function rather than a fixed utility function. Lets speak abstractly for the moment, and consider the space of all relevant utility functions (that is, all utility functions that would change the utility evaluate of an action). At each time step, we now will associate a probability of you transitioning from your current utility function to any of these other utility functions. For any given future state then, we can compute the expected utility. When you run your optimization algorithm to determine your action, what you therefore do is try and maximize the expected utility function, not the current utility function. So the key is going to wind up being assigning estimates to the probability of switching to any other utility function. Doing this in an entirely complete way is difficulty I'm sure, but my guess is that you can come to reasonable estimates that make it possible to do the reasoning.

I think you may be partitioning things that need not necessarily be partitioned and it's important to note that. In the nicotine example (or the "lock the refrigerator door" example in the cited material), this is not necessarily a competition between the wants of different agents. This apparent dichotomy can also be resolved by internal states as well as utility discount factors.

To be specific, revisit the nicotine problem. When a person decides to quit they may not be suffering any discomfort so the utility of smoking at that moment is small. Instead then, the eventual utility of longer life wins out and the agent decides to stop smoking. However, once discomfort sets in, it combines with the action of smoking because smoking will relieve the discomfort. Now the individual still has the other utility assigned to not dying sooner (which would favor the "don't smoke" action). However, the death outcome will happen much later. Even though death is far worse than the current discomfort being felt (assuming a "normal" agent ;), so long as the utilities also operate on a temporal discount factor, that utility may be reduced to be smaller than the utility of smoking that will remove the current discomfort due to how much it gets discounted from it happen much further in the future.

At no point have we needed to postulate that these are separate competing agents with different wants and this seeming contradiction is still perfectly resolved with a single utility function. In fact, wildly different agent behavior can be revealed by mere changes in the discount factor for enumerable reinforcement learning (RL) agents where discount and reward functions are central to the design of the algorithm.

Now, which answer to the question is true? Is the smoke/don't smoke contradiction a result of competing agents or discount factors and internal states? I suppose it could be either one, but it's important to not assume that these examples directly indicate that there are competing agents with different desires, otherwise you may lose yourself looking for something that isn't there.

Of course, even if we assume that there are competing agents with different desires, it seems to me this still can be, at least mathematically, reduced to a single utility function. All it means, is that you apply weights to the utilities of different agents, and then standard reasoning mechanisms are employed.