An AI is going to tell us what we would want, if only we knew more. Apparently, there is an assumption here that the AI knows things we don't. Personally, I worry a bit that an AI will come to believe things that are not true. In fact, I worry about it most when the AI claims to know something that mankind does not know - something dealing with human values. Why do I worry about that? Something someone wrote somewhere presumably. But maybe that is not the kind of superior AI 'knowledge' that Eliezer is talking about here.

Rebuttal: Most people in the world believe in a religion that is wrong. (This conclusion holds regardless of which, if any, world religion happens to be true.) Would we want an A.I. that enforces the laws of a false religion because people want the laws of their religion enforced? (Assume that people would agree that the AI shouldn't enforce the laws of false religions.)

If Fred would adamantly refuse to even consider the possibility that box B contains a diamond, while also adamantly refusing to discuss what should happen in the event that he is wrong in this sort of case, and yet Fred would still be indignant and bewildered on finding that box A is empty, Fred's volition on this problem is muddled.

Am I alone in preferring, in this situation, that the AI not diagnose a 'muddle', and instead give Fred box A after offering him the relevant knowledge?

What if box A actually contains a bomb that explodes when Fred opens it? Should the AI still give Fred the box?