This statement bugs me because I don't see that any solution to that problem has been developed within LW's avantgarde decision theories. In fact, they often introduce self-referential statements, so this problem in a way should be more pressing for them. The "AIXI-like agent" just doesn't have a self-representation; but a LWDT (LessWrong decision theory) agent, insofar as its decision theory revolves around self-referential propositions, does need a capacity for self-representation, and yet I don't remember this problem being discussed very much. It's more as if the problem has been overlooked because so much of the theory is discussed in natural language, and so the capacity for self-referential semantics that natural-language statements provide has been taken for granted.
There is actually a decades-old body of work in computer science on self-representation, for example under the name of "computational reflection"; it was the subject of the thesis of Pattie Maes at MIT. But there's no magic bootstrap here, whereby ordinary reference turns into self-reference. It's just a matter of coding up, by hand, structures that happen to represent aspects of themselves, and then giving them causal connections so that those representations co-vary appropriately with the things they represent. This would appear to be an adequate solution to the problem, and I don't see that any alternative solutions have been invented locally.
A UDT agent does not need to be explicitly told how to represent itself, besides knowing its own code.
This is because a UDT agent does not make an attempt to explicitly control a particular instantiation of itself. A UDT agent looks at the world and tries to control the entire thing by controlling logical facts about its own behavior. If the world happens to contain patterns similar to the agent, then the agent will recognize that controlling its own output will control the behavior of those patterns. The agent could also infer that by destroying those pa...
Daniel Dewey, 'Learning What to Value'
Abstract: I.J. Good's theory of an "intelligence explosion" predicts that ultraintelligent agents will undergo a process of repeated self-improvement. In the wake of such an event, how well our values are fulfilled will depend on whether these ultraintelligent agents continue to act desirably and as intended. We examine several design approaches, based on AIXI, that could be used to create ultraintelligent agents. In each case, we analyze the design conditions required for a successful, well-behaved ultraintelligent agent to be created. Our main contribution is an examination of value-learners, agents that learn a utility function from experience. We conclude that the design conditions on value-learners are in some ways less demanding than those on other design approaches.