Nice point. I think most of the time this doesn't apply, because agents don't just try to maximize "utility" as some abstract quantity, but possess (transparently or opaquely) some function which they concretely try to increase. Utility is a device, specific to each agent, that refers to states of the world, and isn't a kind of universal brownie point that everyone likes.
On the other hand, putting "utility" as your utility is fertile ground for wireheading temptations. I define my utility to be ! There, I win.
(I used this paper as a reference for functional decision theory (FDT), which is essentially an improved version of timeless decision theory and updateless decision theory)
This post is a reflection on decision processes that refer to themselves (call them introspective agents if you like) with a view toward investigating the problem of counterfactuals. It ruminates on the fact that logical uncertainty about the output of a decision process is often a good thing for agents.
Let's begin with a tentative definition. A deterministic agent is autonomous with respect to if cannot predict . has logical uncertainty about 's actions. This occurs, for instance, in Newcomb's problem, when the... (read 597 more words →)
Thanks. I made some quick changes but it's probably not structured enough for your taste. Ah well