A putative new idea for AI control; index here.

It's occurred to me that there is a framework where we can see all "indifference" results as corrective rewards, both for the utility function change indifference and for the policy change indifference.

Imagine that the agent has reward $R_0$ and is following policy ${\pi }_0$, and we want to change it to having reward $R_1$ and following policy ${\pi }_1$.

Then the corrective reward we need to pay it, so that it doesn't attempt to resist or cause that change, is simply the difference between the two expected values:

• $V\left(R_0|{\pi }_0\right)-V\left(R_1|{\pi }_1\right)$,

where $V$ is the agent's own valuation of the expected reward, conditional on the policy.

This explains why off-policy reward-based agents are already safely interruptible: since we change the policy, not the reward, $R_0=R_1$. And since off-policy agents have value estimates that are indifferent to the policy followed, $V\left(R_0|{\pi }_0\right)=V\left(R_1|{\pi }_1\right)$, and the compensatory rewards are zero.