The point is that the real me knows that 'simulated me' is going to be different enough from real me, as to be an entirely different being which has in fact little in common with me. There's bigger problem, the AI could simulate beings, and then torture them. But that AI would need a ton of computing power.

I think what would happen in the real world, is that nobody would believe that the AI is intelligent unless AI really goes out of the way to convince everyone that it is intelligent.