Stuart_Armstrong comments on AIXI and Existential Despair - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (37)
The point is that a physical simulation of the universe includes a description of A. If A simulates the universe, it sees that its outputs appear on W2 without postulating anything beyond physical law. (This is all most straightforward in the case where A does perfect induction and the shortest explanation for A's observations takes the form of pointing to A in the universe, which is the one I started with; if A is capable of executing human-equivalent reasoning, though, the same thing happens in less easily formally analyzed universes.) I will try to edit the post to be more clear.
A true AIXI can't model itself at all. A real AIXI variant (such as AIXItl) will not be able to model itself except in the rarest circumstances. Therefore its model of the universe as containing A is not going to be a simple one, because it can't model A's outputs. Therefore it seems that it will always be unable (as you said in another comment) to "recompute its inputs from the environment".
I can't model myself (or other humans) at all either, and yet I can still learn a model in which human behavior results from physical law, and make predictions on the basis of that model.