'Utility Indifference' (2010) by FHI researcher Stuart Armstrong

lukeprog

I just noticed that LessWrong has not yet linked to FHI researcher Stuart Amstrong's brief technical report, Utility Indifference (2010). It opens:

Consider an AI that follows its own motivations. We're not entirely sure what its motivations are, but we would prefer that the AI cooperate with humanity; or, failing that, that we can destroy it before it defects. We'll have someone sitting in a room, their finger on a detonator, ready at the slightest hint of defection.

Unfortunately as has been noted... this does not preclude the AI from misbehaving. It just means that the AI must act to take control of the explosives, the detonators or the human who will press the button. For a superlatively intelligence AI, this would represent merely a slight extra difficulty. But now imagine that the AI was somehow indifferent to the explosives going off or not (but that nothing else was changed). Then if ever the AI does decide to defect, it will most likely do so without taking control of the explosives, as that would be easier than otherwise. By "easier" we mean that the chances of failure are less, since the plan is simpler... recall that under these assumptions, the AI counts getting blown up as an equal value to successfully defecting. How could we accomplish this indifference?

I just noticed that LessWrong has not yet linked to FHI researcher Stuart Amstrong's brief technical report, Utility Indifference (2010). It opens:

Consider an AI that follows its own motivations. We're not entirely sure what its motivations are, but we would prefer that the AI cooperate with humanity; or, failing that, that we can destroy it before it defects. We'll have someone sitting in a room, their finger on a detonator, ready at the slightest hint of defection.

Unfortunately as has been noted... this does not preclude the AI from misbehaving. It just means that the AI must act to take control of the explosives, the detonators or the human who will press the button. For a superlatively intelligence AI, this would represent merely a slight extra difficulty. But now imagine that the AI was somehow indifferent to the explosives going off or not (but that nothing else was changed). Then if ever the AI does decide to defect, it will most likely do so without taking control of the explosives, as that would be easier than otherwise. By "easier" we mean that the chances of failure are less, since the plan is simpler... recall that under these assumptions, the AI counts getting blown up as an equal value to successfully defecting. How could we accomplish this indifference?

This article jumps right into probabilistic explanation without actually putting into English what it's trying to achieve. It seems like the idea is to set its function such that (say) it is indifferent to whether or not it gets blown up, regardless of the other consequences. This ensures that it's always possible to blow it up, as it never cares about that, and will completely ignore you if you try - even if it becomes aware that you're blowing it up, it won't care about it, and won't stop you.

The problem is how to frame its reward function. Something like - if you're blown up, your reward for being blown up is precisely your current expected reward that you think you would have got if you'd carried on as you were.

The problem of this indifference is that blowing yourself up is almost certainly an easier problem than what we wanted the AI to do - it has to be, otherwise we wouldn't be able to figure out how to blow up the AI. The AI ought to conclude that it might as well blow itself up because it doesn't make any difference to do so. There's no reason why it should do this, but equally, no reason why not.....

It's more subtle than that: the AI will assume that every attempt to blow it up (in that specific way) will fail. Or rather, it will behave as if it believed that attempt will fail. This is because the utility of "successful blow up" is reset to that of "unsucessful blow up", rather than the other way round. So it's equally happy to be blown up now, as it was before under the assumption it wouldn't be blown up.

Since it's behaving as if it believed the blow up would fail, it will only trigger the blow-up deliberately if there is another good reason for it to do so.

4

'Utility Indifference' (2010) by FHI researcher Stuart Armstrong

4

4

4

'Utility Indifference' (2010) by FHI researcher Stuart Armstrong

4

4