not all choices correspond to maximizing such a function - any time choices go in a circle, for instance, you're not maximizing a function. We could imagine a very simple machine with a 3-state memory. It wants to go from A to B, and from B to C, and from C to A. Its choices are always a function if its internal state. But its choices don't maximize a function of its internal state.

Here's the corresponding utility function - assuming that state transitions are tied to actions.

• If IAM(A) { U(A) = 0, U(B) = 1 U(C) = 0; }
• If IAM(B) { U(A) = 0, U(B) = 0 U(C) = 1; }
• If IAM(C) { U(A) = 1, U(B) = 0 U(C) = 0; }

Using simple maximisation algorithms (e.g. gradient descent) on that utility landscape will produce the behaviour in question. More sophisticted algorithms will do no better.

For one thing, the agent may not believe what you say.

Okay. Replace "offer it a choice" with "offer it a choice, and provide sufficient Bayesian evidence that this is this choice faced." This doesn't lead anywhere anyhow.

Your "BartlebeyBot" agent totally ignored Bayesian evidence. By what rule does "my" example agent have to listen and respond to such evidence, while "yours" does not? Again, I don't think your proposed counter example is remotely convincing.

Why do you think there's a counter-example? Did you read the referenced Dewey paper about O-Maximisers?

So: the only information available to any agent is in the form of its internal state and its sensory channels. Any function it computes must have that domain (or some subset of it). Confining the agent to that domain isn't any kind of restriction. All utility functions calulated over the state of the world necessarily correspond to other utility functions calulated over the domain of internal state and sensory input.

Your example seems wrong to me. The problem is with:

For example, you could offer to change a robot's sensory contents and internal state to something with higher utility than its current state - and if the agent refuses, you will reset it. If we were using a "utility wrapper" model, all modeled agents would say yes.

That's not correct. For one thing, the agent may not believe what you say.

Your comment doesn't seem very clear to me. Are you thinking that a "utility function" needs to have a specific domain which is not simply sensory contents and internal state? If so, do you have a reference for that notion?

The term "function" - as used on the page - is a technical term with a clearly-established meaning.

That is not a problem. A compact utility-based description of an agent's behaviour is only ever slightly longer than the shortest description of it available. It's easy to show that by considering a utility-based "wrapper" around the shortest description.

Yes.

This definition has the important feature of restricting "Friendly AI" to designs that have a utility function.

That doesn't seem important - for the reason described here - where it says:

Utility maximisation is a general framework which is powerful enough to model the actions of any computable agent. The actions of any computable agent - including humans - can be expressed using a utility function.

I recommend that most people at least try melatonin.

I wanted to write about my opinion that human values can't be divided into final values and instrumental values, the way discussion of FAI presumes they can. This is an idea that comes from mathematics, symbolic logic, and classical AI. A symbolic approach would probably make proving safety easier. But human brains don't work that way. You can and do change your values over time, because you don't really have terminal values.

You may have wanted to - but AFAICS, you didn't - apart from this paragraph. It seems to me that it fails to make its case. The split applies to any goal-directed agent, irrespective of implemetation details.

Probably Adaptation in Natural and Artificial Systems. Here's Holland's (most famous theorem in the area](http://en.wikipedia.org/wiki/Holland%27s_schema_theorem). It doesn't suggest genetic algorithms make for some kind of optimal search - indeed, classical genetic algorithms are a pretty stupid sort of search.