So the AI is not an expected utility maximizer?
If it is not, then what is it? If it is, then what calculations did it use to reach the above decision - what were the assigned probabilities to the scenarios mentioned?
So the AI is not an expected utility maximizer?
It's an expected utility maximizer, but it considers the expected utility of its decision process, not just the expected utility of individual decisions. In a world where there exist more known liars than known superhuman entities, and any liar can claim superhuman powers, any decision process that allows them to exploit you is of negative expected utility.
It's like the professor who in the example agrees to accept a delayed essay that was delayed for the reason of a grandmother's death, because this is a ...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.