A putative new idea for AI control; index here.

A simple way of thinking that I feel clarifies a lot of issues (related to Blue Minimising Robot):

Suppose you have an entity $H$ that follows algorithm $a{l}^H$. Then define:

• What $H$ does is its actions/outputs in the environment.
• What $H$ is is $a{l}^H$.
• What $H$ wants is an interpretation of what $H$ does (and possibly what it is), in order to construct a utility function or reward function corresponding with its preferences.

The interpretation part of wants is crucial, but it is often obscured in practice in value learning. That's because we often start with things like `$H$ is a boundedly rational agent that maximises $u$...', or we lay out the agent in such a way that that's clearly the case.

What we're doing there is writing the entity as $a{l}^H\left(u\right)$ --- an algorithm with a special variable $u$ that tracks what the entity wants. In the case of cooperative inverse reinforcement learning, this is explicit, as the human's values are given by a $\theta$, known to the human. Thus the human's true algorithm is $a{l}^H(\cdot )$, the human observes $\theta$, meaning that $\theta$ is an objective fact about the universe. And then the human follows $a{l}^H\left(\theta \right)$.

Note here that knowing what the human is in the one-variable sense (i.e. knowing $a{l}^H(\cdot )$) helps with the correct deduction about what they want - while simply knowing the joint $a{l}^H\left(\theta \right)$ does not.

In contrast an interpretation starts with a zero-variable algorithm, and attempts to construct a one-variable one. There for, given $a{l}^H$ it constructs (one or more) $a{l}_i^H(\cdot )$ and $u_i$ such that

• $a{l}^H=a{l}_i^H\left(u_i\right)$.

This illustrates the crucial role of interpretation, especially if $a{l}^H$ is highly complex.