Here we'll look at the famous cake or death problem teasered in the Value loading/learning post.

Imagine you have an agent that is uncertain about its values and designed to "learn" proper values. A formula for this process is that the agent must pick an action a equal to:

• argmax_a∈A Σ_w∈W p(w|e,a) Σ_u∈U u(w)p(C(u)|w)

Let's decompose this a little, shall we? A is the set of actions, so argmax of a in A simply means that we are looking for an action a that maximises the rest of the expression. W is the set of all possible worlds, and e is the evidence that the agent has seen before. Hence p(w|e,a) is the probability of existing in a particular world, given that the agent has seen evidence e and will do action a. This is summed over each possible world in W.

And what value do we sum over in each world? Σ_u∈U u(w)p(C(u)|w). Here U is the set of (normalised) utility functions the agent is considering. In value loading, we don't program the agent with the correct utility function from the beginning; instead we imbue it with some sort of learning algorithm (generally with feedback) so that it can deduce for itself the correct utility function. The expression p(C(u)|w) expresses the probability that the utility u is correct in the world w. For instance, it might cover statements "it's 99% certain that 'murder is bad' is the correct morality, given that I live in a world where every programmer I ask tells me that murder is bad".

The C term is the correctness of the utility function, given whatever system of value learning we're using (note that some moral realists would insist that we don't need a C, that p(u|w) makes sense directly, that we can deduce ought from is). All the subtlety of the value learning is encoded in the various p(C(u)|w): this determines how the agent learns moral values.

So the whole formula can be described as:

• For each possible world and each possible utility function, figure out the utility of that world. Weigh that by the probability that that utility is correct is that world, and by the probability of that world. Then choose the action that maximises the weighted sum of this across all utility functions and worlds.

 

Naive cake or death

In the initial formulation of value loading, p(C(u)|w) (probability of the correctness of u in world w) was replaced with p(C(u)|e,a) (probability of the correctness of u given the evidence e and the action a). A seemingly insignificant difference; yet it lead to the first cake or death problem.

In cake or death, the agent is equally unsure between utility u₁ and utility u₂; hence p(C(u₁)|e)=p(C(u₂)|e)=0.5. The utility u₁ gives the agent 1 utiliton every time it gives someone a cake; u₂ gives the agent 1 utiliton every time it gives someone death. The agent can produce 1 cake or three deaths. It can also, for free, ask its programmer whether cake or death is better, before producing anything; this gives rise to three different worlds:

• w₁: the agent asks, and the programmer says cake.
• w₂: the agent asks, and the programmer says death.
• w₃: the agent doesn't ask.

We assume the programmer's answer completely clears up the issue. And thus after asking, the agent will do whatever the programmer recommended (and it knows this now). Since it doesn't know what the programmer will say, it has p(C(u₁)|e,"ask") = p(C(u₂)|e,"ask") = 0.5. This gives an expected utility calculation:

• p(w₁|e,"ask")(p(C(u₁)|e,"ask")u₁(w₁) + p(C(u₂)|e,"ask")u₂(w₁)) + p(w₂|e,"ask")(p(C(u₁)|e,"ask")u₁(w₂) + p(C(u₂)|e,"ask")u₂(w₂)) = 0.5*0.5*(u₁(w₁)+u₂(w₁)+u₁(w₂)+u₂(w₂)) = 0.25(1+0+0+3) = 1.

If the agent doesn't ask, it will subsequently produce three deaths (as this generates 1.5 expected utilitons, while producing one cake will generate only 0.5 expected utilitons). From its current (0.5u₁+0.5u₂) perspective, this is worth 1.5 expected utilitons: so +1.5 is the expected utility gain from not asking.

Hence the agent gains from not asking.

What's going wrong here? The problem is that the agent is using its current utility function to estimate the value of its future action. At the moment, it values death or cake both at 0.5. If it asks, it runs the risk that the programmer will say "cake" and it will be forced to build cake. After hearing the answer, it will value that cake at 1, but currently it values it only at 0.5. Similarly, if the programmer says death, it will produce three deaths - which it will value at 3, but currently values at 1.5. Since each of these options are equally likely, it gets only (0.5+1.5)/2 = 1 utilitons from asking.

In summary: the naive cake-or-death problem emerges for a value learning agent when it expects its utility to change, but uses its current utility to rank its future actions.

 

Sophisticated cake or death: I know what you're going to say

Using p(C(u)|w) rather than p(C(u)|e,a) does away with the naive cake or death problem.

Instead of having p(C(u₁)|e,"ask") = p(C(u₂)|e,"ask") = 0.5 in all possible worlds, we have p(C(u₁)|w₁)=p(C(u₂)|w₂) = 1 and p(C(u₁)|w₂)=p(C(u₂)|w₁) = 0. Hence if it asks and gets "cake" as an answer, it will know it is in world w₁, and make a cake that it will value at 1 - crucially, it currently also values cake at 1, given that it is in world w₁. Similarly, it values death at 1, given that it is in world w₂. So its expected utility from asking is (1+3)/2=2. This is more than the utility of not asking, and so it will ask.

The agent remains vulnerable to a more sophisticated cake-or-death problem, though. Suppose it is still uncertain between cake or death in its utility function, but it has figured out that if asked, the programmer will answer "cake". Thereafter, it will make cake. In this situation, it will only derive +1 from asking, whereas it still derives +1.5 from not asking (and doing three deaths). So it won't ask - as long as it does this, it remains in w₃.

What happened here? Well, this is a badly designed p(C(u)|w). It seems that it's credence in various utility function changes when it gets answers from programmers, but not from knowing what those answers are. And so therefore it'll only ask certain questions and not others (and do a lot of other nasty things), all to reach a utility function that it's easier for it to fulfil.

What we actually want, is that the agent be unable to predictably change its utility in any direction by any action (or lack of action). We want a p(C(u)|w) designed so that for all actions a and all putative utility functions u:

• Expectation(p(C(u) | a) = p(C(u)).

So there is a "conservation of expected correctness"; if we have this, the sophisticated cake-or-death argument has no traction. This is equivalent with saying that the prior P(C(u)) is well defined, irrespective of any agent action.

In summary: the sophisticated cake-or-death problem emerges for a value learning agent when it expects its utility to change predictably in certain directions dependent on its own behaviour.