= 27d567bc2d48d9f40e9ccc20ea370b15)
Cyan comments on The Problem with AIXI - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (78)
My kids would have long since have been maimed or killed by exactly that kind of mistake, if not for precautions taken by and active monitoring by their parents.
Yeah, that's right. Having a naturalized architecture may be necessary for general intelligence concerning self-modifications, even if it's not sufficient. Other things are necessary too, like large, representative data sets.
If AIXI starts off without a conception of death but eventually arrives at one, then the criticism of AIXI I've been making is very wrong. The key question is whether AIXI ever grows up into a consistently rational agent.
I can't actually understand/grok/predict what it is like to not exist, but I know that if I die, I will not learn or act anymore. That seems to be all that naturalized reasoning can give me, and all that is necessary for an AI too.
A naturalized agent's hypotheses can be about world-states that include the agent, or world-states that don't include the agent. A Cartesian agent's hypotheses are all about the agent's internal states, and different possible causes for those states, so the idea of 'world-states that don't include the agent' can't be directly represented. Even a halting program in AIXI's hypothesis space isn't really a prediction about how a world without AIXI would look; it's more a prediction about how Everything (including AIXI) could come to an end.
Our ultimate goal in building an AI isn't to optimize the internal features of the AI; it's to optimize the rest of the world, with the AI functioning as a tool. So it seems likely that we'll want our AI's beliefs to look like pictures of an objective world (in which agents like the AI happen to exist, sometimes).
A sequence predictor's predictions are all about the agent's input tape states*, and different possible causes for those states. The hypotheses are programs that implement entire models of the Universe, and these can definitely directly represent world-states which don't include the agent.
* More realistically, the states of the registers where the sensor data is placed.
ETA: I wonder if this intuition is caused by that fact that I am a practicing Bayesian statistician, so the distinction between posterior distributions and posterior predictive distributions is more salient to me.
The analogy is made somewhat more precise by my new formalism.