Interesting-- the failure mode that occurred to me is a paper-clipper which is designed to prefer virtual paper clips, so it turns the earth/the solar system/ the lightcone into computronium to run virtual paperclips.

If defining stay in the box is that hard, I'm not feeling hopeful about the possibility of defining protect humans.