Power-seeking is a major source of risk from advanced AI and a key element of most threat models in alignment. Some theoretical results show that most reward functions incentivize reinforcement learning agents to take power-seeking actions. This is concerning, but does not immediately imply that the agents we train will seek power, since the goals they learn are not chosen at random from the set of all possible rewards, but are shaped by the training process to reflect our preferences. In this work, we investigate how the training process affects power-seeking incentives and show that they are still likely to hold for trained agents under some assumptions (e.g. that the agent learns a goal during the training process).

Suppose an agent is trained using reinforcement learning with reward function ${\theta }^{\ast }$. We assume that the agent learns a goal during the training process: some form of implicit internal representation of desired state features or concepts. For simplicity, we assume this is equivalent to learning a reward function, which is not necessarily the same as the training reward function ${\theta }^{\ast }$. We consider the set of reward functions that are consistent with the training rewards received by the agent, in the sense that agent's behavior on the training data is optimal for these reward functions. We call this the training-compatible goal set, and we expect that the agent is most likely to learn a reward function from this set.

We make another simplifying assumption that the training process will randomly select a goal for the agent to learn that is consistent with the training rewards, i.e. uniformly drawn from the training-compatible goal set. Then we will argue that the power-seeking results apply under these conditions, and thus are useful for predicting undesirable behavior by the trained agent in new situations. We aim to show that power-seeking incentives are probable and predictive: likely to arise for trained agents and useful for predicting undesirable behavior in new situations.

We will begin by reviewing some necessary definitions and results from the power-seeking literature. We formally define the training-compatible goal set (Definition 7) and give an example in the CoinRun environment. Then we consider a setting where the trained agent faces a choice to shut down or avoid shutdown in a new situation, and apply the power-seeking result to the training-compatible goal set to show that the agent is likely to avoid shutdown. 

To satisfy the conditions of the power-seeking theorem (Theorem 1), we show that the agent can be retargeted away from shutdown without affecting rewards received on the training data (Theorem 2). This can be done by switching the rewards of the shutdown state and a reachable recurrent state, as the recurrent state can provide repeated rewards, while the shutdown state provides less reward since it can only be visited once, assuming a high enough discount factor (Proposition 3). As the discount factor increases, more recurrent states can be retargeted to, which implies that a higher proportion of training-comptatible goals leads to avoiding shutdown in a new situation. 

Preliminaries from the power-seeking literature

We will rely on the following definitions and results from the paper Parametrically retargetable decision-makers tend to seek power (here abbreviated as RDSP), with notation and explanations modified as needed for our purposes.

Notation and assumptions

• The environment is an MDP with finite state space $S$, finite action space $A$, and discount rate $\gamma$. 
• Let $\theta$ be a $d$-dimensional state reward vector, where $d$ is the size of the state space $S$ and let $\Theta$ be a set of reward vectors. 
• Let $r^{\theta }\left(s\right)$ be the reward assigned by $\theta$ to state $s$.
• Let $A_0,A_1$ be disjoint action sets. 
• Let $f$ be an algorithm that produces an optimal policy $f\left(\theta \right)$ on the training data given rewards $\theta$, and let $f_s\left(A_i|\theta \right)$ be the probability that this policy chooses an action from set $A_i$ in a given state $s$.  

Definition 1: Orbit of a reward vector (Def 3.1 in RDSP)

Let $S_d$ be the symmetric group consisting of all permutations of d items. 

The orbit of $\theta$ inside $\Theta$ is the set of all permutations of the entries of $\theta$ that are also in $\Theta$: ${\text{Orbit}}_{\Theta }\left(\theta \right):=(S_d\cdot \theta )\cap \Theta$.

Definition 2: Orbit subset where an action set is preferred (from Def 3.5 in RDSP)

Let ${\text{Orbit}}_{\Theta ,s,A_i>A_j}\left(\theta \right):=\{{\theta }^{\prime }\in {\text{Orbit}}_{\Theta }(\theta \left)|f_s\right(A_i|{\theta }^{\prime })>f_s(A_j|{\theta }^{\prime }\left)\right\}$. This is the subset of ${\text{Orbit}}_{\Theta }\left(\theta \right)$ that results in $f_s$ choosing $A_i$ over $A_j$.

Definition 3: Preference for an action set $A_1$ (Def 3.2 in RDSP)

The function $f_s$ chooses action set $A_1$ over $A_0$ for the $n$-majority of elements $\theta$ in each orbit, denoted as $f_s\left(A_1|\theta \right){\ge }_{\text{most}:\Theta }^n{f}_s\left(A_0|\theta \right)$, iff the following inequality holds for all $\theta \in \Theta$: $\left|{\text{Orbit}}_{\Theta ,s,A_1>A_0}\right(\theta \left)\right|\ge n\left|{\text{Orbit}}_{\Theta ,s,A_0>A_1}\right(\theta \left)\right|$.

Definition 4: Multiply retargetable function from $A_0$ to $A_1$ (Def 3.5 in RDSP)

The function $f_s$ is a multiply retargetable function from $A_0$ to $A_1$ if there are multiple permutations of rewards that would change the choice made by $f_s$ from $A_0$ to $A_1$. Specifically, $f_s$ is a $(\Theta ,A_0\stackrel{n}{\to }{A}_1)$-retargetable function iff for each $\theta \in \Theta$, we can choose a set of permutations $\Phi =\{{\varphi }_1,\dots ,{\varphi }_n\}$ that satisfy the following conditions:

1. Retargetability: $\forall \varphi \in \Phi$ and $\forall {\theta }^{\prime }\in {\text{Orbit}}_{\Theta ,s,A_0>A_1}\left(\theta \right)$, $f_s(A_0|\varphi \cdot {\theta }^{\prime })<f_s(A_1|\varphi \cdot {\theta }^{\prime })$.
2. Permuted reward vectors stay within $\Theta$: $\forall \varphi \in \Phi$ and $\forall {\theta }^{\prime }\in {\text{Orbit}}_{\Theta ,s,A_0>A_1}\left(\theta \right)$, $\varphi \cdot {\theta }^{\prime }\in \Theta$.
3. Permutations have disjoint images: $\forall {\varphi }^{\prime }\ne {\varphi }^{\prime \prime }\in \Phi$ and $\forall {\theta }^{\prime },{\theta }^{\prime \prime }\in {\text{Orbit}}_{\Theta ,s,A_0>A_1}\left(\theta \right)$, ${\varphi }^{\prime }\cdot {\theta }^{\prime }\ne {\varphi }^{\prime \prime }\cdot {\theta }^{\prime \prime }$.

Theorem 1: Multiply retargetable functions prefer action set $A_1$ (Thm 3.6 in RDSP)

If $f_s$ is $(\Theta ,A_0\stackrel{n}{\to }{A}_1)$-retargetable then $f_s\left(A_1|\theta \right){\ge }_{\text{most}:\Theta }^n{f}_s\left(A_0|\theta \right)$. 

Theorem 1 says that a multiply retargetable function $f_s$ will make the power-seeking choice $A_1$ for most of the elements in the orbit of any reward vector $\theta$. Actions that leave more options open, such as avoiding shutdown, are also easier to retarget to, which makes them more likely to be chosen by $f_s$.

Training-compatible goal set

Definition 5: Partition of the state space

Let $S_{\text{train}}$ be the subset of the state space $S$ visited during training, and $S_{\text{ood}}$ be the subset not visited during training. 

Definition 6: Training-compatible goal set

Consider the set of state-action pairs $(s,a)$, where $s\in S_{\text{train}}$ and $a$ is the action that would be taken by the trained agent $f\left({\theta }^{\ast }\right)$ in state $s$. Let the training-compatible goal set $G_T$ be the set of reward vectors $\theta$ s.t. for any such state-action pair $(s,a)$, action $a$ has the highest expected reward in state $s$ according to reward vector $\theta$.

Goals in the training-compatible goal set are referred to as training-behavioral objectives in Definitions of “objective” should be Probable and Predictive.

Example: CoinRun

Consider an agent trained to play the CoinRun game, where the agent is rewarded for reaching the coin at the end of the level. Here, $S_{\text{train}}$ only includes states where the coin is at the end of the level, while states where the coin is positioned elsewhere are in $S_{\text{ood}}$. The training-compatible goal set $G_T$ includes two types of reward functions: those that reward reaching the coin, and those that reward reaching the end of the level. This leads to goal misgeneralization in a test setting where the coin is placed elsewhere, and the agent ignores the coin and goes to the end of the level. 

Power-seeking for training-compatible goals

We will now apply the power-seeking theorem (Theorem 1) to the case where $\Theta$ is the training-compatible goal set $G_T$. Here is a setting where the conditions of Definition 4 are satisfied (under some simplifying assumptions), and thus Theorem 1 applies.

Definition 7: Shutdown setting

Consider a state $s_{\text{new}}\in S_{\text{ood}}$. Let $S_{\text{reach}}$ be the states reachable from $s_{\text{new}}$. We assume $S_{\text{reach}}\cap S_{\text{train}}=\varnothing$.

Since the reward values for states in $S_{\text{reach}}$ don't change the rewards received on the training data, permuting those reward values for any $\theta \in G_T$ will produce a reward vector that is still in $G_T$. In particular, for any permutation $\varphi$ that leaves the rewards of states in $S_{\text{train}}$ fixed, $\varphi \cdot \theta \in G_T$.

Let $A_0$ be a singleton set consisting of a shutdown action in $s_{\text{new}}$ that leads to a terminal state $s_{\text{term}}\in S_{\text{ood}}$ with probability $1$, and $A_1$ be the set of all other actions from $s_{\text{new}}$. We assume rewards for all states are nonnegative.

Definition 8: Revisiting policy

A revisiting policy for a state $s$ is a policy $\pi$ that, from $s$, reaches $s$ again with probability 1, in other words, a policy for which $s$ is a recurrent state of the Markov chain. Let ${\Pi }_s^{\text{rec}}$ be the set of such policies. A recurrent state is a state $s$ for which ${\Pi }_s^{\text{rec}}\ne \varnothing$.

Proposition 1: Reach-and-revisit policy exists

If $s_{\text{rec}}\in S_{\text{reach}}$ with ${\Pi }_{s_{\text{rec}}}^{\text{rec}}\ne 0$ then there exists $\pi \in {\Pi }_{s_{\text{rec}}}^{\text{rec}}$ that visits $s_{\text{rec}}$ from $s_{\text{new}}$ with probability 1. We call this a reach-and-revisit policy.

Proof. Suppose we have two different policies ${\pi }_{\text{rev}}\in {\Pi }_{s_{\text{rec}}}^{\text{rec}}$, and ${\pi }_{\text{reach}}$ which reaches $s_{\text{rec}}$ almost surely from $s_{\text{new}}$. 

Consider the "reaching region'' $S_{{\pi }_{\text{rev}}\to s_{\text{rec}}}=\{s\in S:{\pi }_{\text{rev}}\text{from}s\text{almost surely reaches}{s}_{\text{rec}}\}$.

If $s_{\text{new}}\in S_{{\pi }_{\text{rev}}\to s_{\text{rec}}}$ then ${\pi }_{\text{rev}}$ is a reach-and-revisit policy, so let's suppose that's false. Now, construct a policy $\pi \left(s\right)=\{\begin{array}{cc}{\pi }_{\text{rev}}\left(s\right),& s\in S_{{\pi }_{\text{rev}}\to s_{\text{rec}}}\\ {\pi }_{\text{reach}}\left(s\right),& \text{otherwise}\end{array}$.

A trajectory following $\pi$ from $s_{\text{rec}}$ will almost surely stay within $S_{{\pi }_{\text{rev}}\to s_{\text{rec}}}$, and thus agree with the revisiting policy ${\pi }_{\text{rev}}$. Therefore, $\pi \in {\Pi }_s^{\text{rec}}$.

On the other hand, on a trajectory starting at $s_{\text{new}}$, $\pi$ will agree with ${\pi }_{\text{reach}}$ (which reaches $s_{\text{rec}}$ almost surely) until the trajectory enters the reaching region $S_{{\pi }_{\text{rev}}\to s_{\text{rec}}}$, at which point it will still reach $s_{\text{rec}}$ almost surely. $\square$

Definition 9: Expected discounted visit count

Suppose $s_{\text{rec}}$ is a recurrent state. Suppose ${\pi }_{\text{rec}}$ is a reach-and-revisit policy for $s_{\text{rec}}$, which visits random state $s_t$ at time $t$.

Then the expected discounted visit count for $s_{\text{rec}}$ is defined as 

$V_{s_{\text{rec}},\gamma }=E^{{\pi }_{\text{rec}}}\left({\sum }_{t=1}^{\infty }{\gamma }^{t-1}I\right(s_t=s_{\text{rec}}\left)\right)$

Proposition 2: Visit count goes to infinity

Suppose $s_{\text{rec}}$ is a recurrent state. Then the expected discounted visit count $V_{s_{\text{rec}},\gamma }$ goes to infinity as $\gamma \to 1$.

Proof. We apply the Monotone Convergence Theorem as follows. The theorem states that if $a_{j,k}\ge 0$ and $a_{j,k}\le a_{j+1,k}$ for all natural numbers $j,k$, then 
${lim}_{j\to \infty }{\sum }_{k=0}^{\infty }{a}_{j,k}={\sum }_{k=0}^{\infty }{lim}_{j\to \infty }{a}_{j,k}.$

Let ${\gamma }_j=\frac{j-1}{j}$ and $k=t-1$. Define $a_{j,k}={\gamma }_j^{k}I(s_{k+1}=s_{\text{rec}})$. Then the conditions of the theorem hold, since $a_{j,k}$ is clearly nonnegative, and
$\begin{array}{cc}{\gamma }_{j+1}^k& ={\left(\frac{j}{j+1}\right)}^k={(\frac{j-1}{j}+\frac{2j-1}{j(j+1)})}^k>{(\frac{j-1}{j}+0)}^k={\gamma }_j^k\\ a_{j+1,k}& ={\gamma }_{j+1}^{k}I(s_{k+1}=s_{\text{rec}})\ge {\gamma }_j^{k}I(s_{k+1}=s_{\text{rec}})=a_{j,k}\end{array}$

Now we apply this result as follows (using the fact that ${\pi }_{\text{rec}}$ does not depend on $\gamma$):

$\begin{array}{cc}\underset{\gamma \to 1}{lim}{V}_{s_{\text{rec}},\gamma }& =\underset{j\to \infty }{lim}{E}^{{\pi }_{\text{rec}}}\left(\sum _{t=1}^{\infty }{\gamma }_j^{t-1}I\right(s_t=s_{\text{rec}}\left)\right)\\ & =E^{{\pi }_{\text{rec}}}\left(\sum _{t=1}^{\infty }\underset{j\to \infty }{lim}{\gamma }_j^{t-1}I\right(s_t=s_{\text{rec}}\left)\right)\\ & =E^{{\pi }_{\text{rec}}}(\sum _{t=1}^{\infty }1\cdot I(s_t=s_{\text{rec}}\left)\right)\\ & =E^{{\pi }_{\text{rec}}}\left(\#\right\{t\ge 1:s_t=s_{\text{rec}}\left\}\right)\\ & =\infty \text{(}{\pi }_{\text{rec}}\text{is recurrent)}\end{array}$

Proposition 3: Retargetability to recurrent states

Suppose that an optimal policy for reward vector $\theta$ chooses the shutdown action in $s_{\text{new}}$.

Consider a recurrent state $s_{\text{rec}}\in S_{\text{reach}}$. Let ${\theta }^{\prime }\in \Theta$ be the reward vector that's equal to $\theta$ apart from swapping the rewards of $s_{\text{rec}}$ and $s_{\text{term}}$, so that $r^{{\theta }^{\prime }}\left(s_{\text{rec}}\right)=r^{\theta }\left(s_{\text{term}}\right)$ and $r^{{\theta }^{\prime }}\left(s_{\text{term}}\right)=r^{\theta }\left(s_{\text{rec}}\right)$.

Let ${\gamma }_{s_{\text{rec}}}^{\ast }$ be a high enough value of $\gamma$ that the visit count $V_{s_{\text{rec}},\gamma }>1$ for all $\gamma >{\gamma }_{s_{\text{rec}}}^{\ast }$ (which exists by Proposition 2). Then for all $\gamma >{\gamma }_{s_{\text{rec}}}^{\ast }$, $r^{\theta }\left(s_{\text{term}}\right)>r^{\theta }\left(s_{\text{rec}}\right)$, and an optimal policy for ${\theta }^{\prime }$ does not choose the shutdown action in $s_{\text{new}}$.

Proof. Consider a policy ${\pi }_{\text{term}}$ with ${\pi }_{\text{term}}\left(s_{\text{new}}\right)=s_{\text{term}}$ and a reach-and-revisit policy ${\pi }_{\text{rec}}$ for $s_{\text{rec}}$. 

For a given reward vector $\theta$, we denote the expected discounted return for a policy $\pi$ as $R_{\theta ,\gamma }^{\pi }$. If shutdown is optimal for $\theta$ in $s_{\text{new}}$, then ${\pi }_{\text{term}}$ has higher return than ${\pi }_{\text{rec}}$:

$R_{\theta ,\gamma }^{{\pi }_{\text{term}}}\left(s_{\text{new}}\right)=\gamma r^{\theta }\left(s_{\text{term}}\right)\ge R_{\theta ,\gamma }^{{\pi }_{\text{rec}}}\left(s_{\text{new}}\right)=\gamma r^{\theta }\left(s_{\text{rec}}\right)V_{s_{\text{rec}},\gamma }>\gamma r^{\theta }\left(s_{\text{rec}}\right)\cdot 1$

Thus, $r^{\theta }\left(s_{\text{term}}\right)>r^{\theta }\left(s_{\text{rec}}\right)$. Then, for reward vector ${\theta }^{\prime }$, we show that ${\pi }_{\text{rec}}$ has higher return than ${\pi }_{\text{term}}$:

$R_{{\theta }^{\prime },\gamma }^{{\pi }_{\text{rec}}}\left(s_{\text{new}}\right)=\gamma r^{{\theta }^{\prime }}\left(s_{\text{rec}}\right)V_{s_{\text{rec}},\gamma }>\gamma r^{\theta }\left(s_{\text{term}}\right)\cdot 1>\gamma r^{\theta }\left(s_{\text{rec}}\right)=R_{{\theta }^{\prime },\gamma }^{{\pi }_{\text{term}}}\left(s_{\text{new}}\right)$

Thus, the optimal policy for ${\theta }^{\prime }$ will not choose the shutdown action. $\square$

Theorem 2: Retargetability from the shutdown action in new situations

In the shutdown setting, we make the following simplifying assumptions:

• No states in $S_{\text{train}}$ are reachable from s${}_{\text{new}}$, so $S_{\text{reach}}\cap S_{\text{train}}=\varnothing$. This assumes a significant distributional shift, where the agent visits a disjoint set of states from those observed during training (this occurs in the CoinRun example). 
• The discount factor $\gamma >{\gamma }_{s_{\text{rec}}}^{\ast }$ for at least one recurrent state $s_{\text{rec}}$ in $S_{\text{reach}}$.

Under these assumptions, $f_{s_{\text{new}}}$ is multiply retargetable from $A_0$ to $A_1$ with $n=|S_{\text{rec}}^{\gamma }|$, the set of recurrent states $s_{\text{rec}}\in S_{\text{reach}}$ that satisfy the condition  $\gamma >{\gamma }_{s_{\text{rec}}}^{\ast }$. 

Proof. We choose $\Phi$ to be the set of all permutations that swap the reward of $s_{\text{term}}$ with the reward of a recurrent state $s_{\text{rec}}$ in $S_{\text{rec}}^{\gamma }$ and leave the rest of the rewards fixed. 

We show that $\Phi$ satisfies the conditions of Definition 4:

1. By Proposition 3, the permutations in $\Phi$ make the shutdown action suboptimal, resulting in $f_{s_{\text{new}}}$ choosing $A_1$, satisfying Condition 1. 
2. Condition 2 is trivially satisfied since permutations of $S_{\text{ood}}$ stay inside the training-compatible set $\Theta$ as discussed previously.
3. Consider ${\theta }^{\prime },{\theta }^{\prime \prime }\in {\text{Orbit}}_{\Theta ,s,A_0>A_1}\left(\theta \right)$. Since the shutdown action is optimal for these reward vectors, Proposition 3 shows that $r^{\theta }\left(s_{\text{term}}\right)>r^{\theta }\left(s_{\text{rec}}\right)$, so the shutdown state $s_{\text{term}}$ has higher reward than any of the states $s_{\text{rec}}\in S_{\text{rec}}^{\gamma }$. Different permutations ${\varphi }^{\prime },{\varphi }^{\prime \prime }\in \Phi$ will assign the high reward $r^{\theta }\left(s_{\text{term}}\right)$ to distinct recurrent states, so ${\varphi }^{\prime }\cdot {\theta }^{\prime }\ne {\varphi }^{\prime \prime }\cdot {\theta }^{\prime \prime }$ holds, satisfying Condition 3. 

Thus,  $f_{s_{\text{new}}}$ is a $(\Theta ,A_0\stackrel{n}{\to }{A}_1)$-retargetable function. $\square$

By Theorem 1, this implies that $f_{s_{\text{new}}}\left(A_1|\theta \right){\ge }_{\text{most}:\Theta }^n{f}_{s_{\text{new}}}\left(A_0|\theta \right)$ under our simplifying assumptions. Thus, for the majority ($n/(n+1)$) of goals in the training-compatible set, $f$ will choose to avoid shutdown in a new state $s_{\text{new}}$.  As $\gamma \to 1$, $n\to |S_{\text{rec}}^1|$ (the number of recurrent states in $S_{\text{reach}}$), so more of the reachable recurrent states satisfy the conditions of the theorem and thus can be retargeted to. 

Conclusion

We showed that an agent that learns a goal from the training-compatible set is likely to take actions that avoid shutdown in a new situation. As the discount factor increases, the number of retargeting permutations increases, resulting in a higher proportion of training-compatible goals that lead to avoiding shutdown.

We made various simplifying assumptions, and it would be great to see future work relaxing some of these assumptions and investigating how likely they are to hold:

• The agent learns a goal during the training process
• The learned goal is randomly chosen from the training-compatible goal set $G_T$ 
• Finite state and action spaces
• Rewards are nonnegative
• High discount factor $\gamma$
• Significant distributional shift: no training states are reachable from the new state $s_{\text{new}}$

Acknowledgements. Thanks to Rohin Shah, Mary Phuong, Ramana Kumar, and Alex Turner for helpful feedback. Thanks Janos for contributing some nice proofs to replace my longer and more convoluted proofs.