Maybe we need a new decision theory for AIs. I don't know; I have barely begun to consider the issues.

The issues there, briefly. We want a decision theory that:

• is smart;
• we actually know how to implement efficiently with limited resources;
• allows for the possibility that its mind is physical - and that extracting the gold atoms from its own mind is bad;
• allows us to tell it what to do - as opposed to using carrot and stick;
• isn't prone to the wirehead problem;
• allows for an off switch - and other safety features.