You don't have to understand the real reason. It just has to convince you. Eliezer Yudkowski can convince someone to let an AI out of a box in a thought experiment, and give him money in real life, despite not believing that to be the logical course of action.

Dead right. It would seem very silly to believe that rationality hits a glass ceiling at human level intelligence. Unlikely though it is, if the AI could predict the number in my head by looking at my facial expressions, then told me to cut my arm off for the good of the human race, I'd suddenly feel very conflicted indeed.

If the AI isn't smart enough to at least come up with a reason I'll accept at face value, something's very wrong. People can be convinced to do incredibly stupid-seeming things for enough money, and if whatever the AI wants is as good for the world as it's supposed to be, there's going to be some way to make money by doing it.

Would an AGI ever try to convince you of something you can't understand? I wouldn't try to explain special relativity to a kindergarten class. Surely an AGI would know perfectly well what you are capable of grasping. If it tries to convince me of something, knowing it cannot, what then are its intentions?