Two things that could very well come out of misunderstandings of the material:
If we have an agent whose actions affect future observations, why can’t we think of information about the agent’s embedding in the environment as being encoded in its observations? For example, in the heating-up game, we could imagine a computer that has sensors that detect heat emanating from its hardware, and that the data from those sensors is incorporated into the input stream of observations of the “environment”. The agent could then learn from past experience that certain
Certainly the agent is ineffective. It destroyed information which could have reduced its value-uncertainty.
7So8res
Thanks for the comments! I'll try to answer briefly.
1. It's useful to think of the task as one of defining the intended intelligence metric. The Legg-Hutter metric can't represent a heating-up game, because there isn't a "heat" channel from the agent to the environment. Is it possible that an agent with high Legg-Hutter intelligence might be able to succeed on the heating up game, given a heat channel? Yes, this is possible. But AIXItl would almost certainly not be able to do this (it cannot consider limiting computation for a few timesteps), and you shouldn't expect agents with high LH score to do this, because this isn't what LH measures. Embedding an agent with high LH score in a heating up game violates an assumption under which the agent was shown to behave well. The problem here is not this one game in particular, the problem is that we still don't know how to define the actual (naturalized) intelligence metric we care about. If we could formalize a set of universes and an embedding rule that allows us to measure agents on problems where their physical embodiment matters, that would constitute progress.
2. You're correct that the agent ultimately needs to choose based purely on information from its observations (well, that and the priors), but there's a difference between agents that are attempting to optimize what they see, and agents that are attempting to optimize what actually happened. Yes, the latter is ultimately an observation-based decision process, but it's a fairly complicated one (note the difficulty of cashing out the word "actually" and the need to worry about the agent's beliefs and their accuracy). The problem of ontology identification is not one of avoiding the fact that the agent must decide based on observations "alone", it's one of figuring out how to build agents that do the specific type of observation-based decision that we prefer (e.g. optimizing for reality rather than sense data). The real question, after all, is "we want agents
Two things that could very well come out of misunderstandings of the material:
... (read more)If we have an agent whose actions affect future observations, why can’t we think of information about the agent’s embedding in the environment as being encoded in its observations? For example, in the heating-up game, we could imagine a computer that has sensors that detect heat emanating from its hardware, and that the data from those sensors is incorporated into the input stream of observations of the “environment”. The agent could then learn from past experience that certain