It's been submitted, but I haven't gotten any word on whether it's accepted yet.

EDIT: Accepted!

This statement bugs me because I don't see that any solution to that problem has been developed within LW's avantgarde decision theories. In fact, they often introduce self-referential statements, so this problem in a way should be more pressing for them. The "AIXI-like agent" just doesn't have a self-representation; but a LWDT (LessWrong decision theory) agent, insofar as its decision theory revolves around self-referential propositions, does need a capacity for self-representation, and yet I don't remember this problem being discussed very much. It's more as if the problem has been overlooked because so much of the theory is discussed in natural language, and so the capacity for self-referential semantics that natural-language statements provide has been taken for granted.

There is actually a decades-old body of work in computer science on self-representation, for example under the name of "computational reflection"; it was the subject of the thesis of Pattie Maes at MIT. But there's no magic bootstrap here, whereby ordinary reference turns into self-reference. It's just a matter of coding up, by hand, structures that happen to represent aspects of themselves, and then giving them causal connections so that those representations co-vary appropriately with the things they represent. This would appear to be an adequate solution to the problem, and I don't see that any alternative solutions have been invented locally.

I really have to read EY's UDT paper. Can you tell me what math prerequisites I need to understand UDT?

Thanks for posting this around! It's great to see it creating discussion.

I'm working on replies to the points you, Bill Hibbard, and Curt Welch have made. It looks like I have some explaining to do if I want to convince you that O-maximizers aren't a subset of reward maximizers-- in particular, that my argument in appendix B doesn't apply to O-maximizers.

Response to Curt Welch:

Sadly, what he seems to have failed to realize, is that any actual implementation of an O-Maximizer or his Value-learners must also be reward maximizerr. Is he really that stupid so as not to understand they are all reward maximizer?

Zing! I guess he didn't think I was going to be reading that. To be fair, it may seem to him that I've made a stupid error, thinking that O-maximizers behave differently than reward maximizers. I'll try to explain why he's mistaken.

A reward maximizer acts so as to bring about universes in which the rewards it receives are maximized. For this reason, it will predict and may manipulate the future actions of its rewarder.

An O-maximizer with utility function U acts so as to bring about universes which score highly according to U. For this reason, it is quite unlikely to manipulate or alter its utility function, unless its utility function directly values universes in which it self-alters.

In particular, note that an O-maximizer does not act so as to bring about universes in which the utility it assigns to the universe is maximized. Where the reward maximizer predicts and "cares about" what the rewarder will say tomorrow, an O-maximizer uses its current utility function to evaluate futures and choose actions.

O-maximizers and reward maximizers have different relationships with their "motivators" (utility function vs. rewarder), and they behave differently when given the option to alter their motivators. It seems clear to me that they are distinct.

The only difference is in the algorithm it uses to calculate the "expected value". Dose he not understand that if you build a machine to do this, that there must be hardware in the machine that calculates that expected value? And that such a machine can then be seen as two machines, one which is calculating the expected value, and the other which is picking actions to maximize the output of that calculation? And once you have that machine, his argument of appendix B once again applies?

Actually trying to apply the argument in Appendix B to an O-maximizer, implemented or in the abstract, using the definitions given in the paper instead of reasoning by analogy, is sufficient to show that this is also incorrect.

An agent of unbounded intelligent will always reach a point of understanding he has the option to try and modify the reward function which means the wirehead problem is always on the table.

It may have the option, but will it be motivated to alter its "reward function"? Consider an O-maximizer with utility function U. It acts to maximize the universe's utility as measured by U. How would the agent's alteration of its own utility function bring about universes that score highly according to U?

Response to Bill Hibbard:

It seems to me that every O-maximizer can be expressed as a reward maximizer. Specifically, comparing equations (2) and (3), given an O-maximizer we can define reward r sub(m) (by this notation I mean "r subscript m") as:

r sub(m) = SUM(r in R) U(r)P(r|yx sub(<=m))

and r sub(i) = 0 for i<m, where the paper sets m to the final time step, following Nick Hay. The reward maximizer so defined will behave identically with the O-maximizer.

In the reward-maximization framework, rewards are part of observations and come from the environment. You cannot define "r sub(m)" to be equal to something mathematically, then call the result a reward-maximizer; therefore, Hibbard's formulation of an O-maximizer as a reward-maximizer doesn't work.

If this is correct, doesn't the "characteristic behavior pattern" shown for reward maximizers in Appendix B, as stated in Section 3.1, also apply to O-maximizers?

Since the construction was incorrect, this argument does not hold.