Okay, I'm clearly not communicating the essential point well enough here.

Didn't you claim in your paper that an AGI will only act correctly if its ontology is sufficiently similar to our own. But what does constitute a sufficiently similar ontology? And where do you draw the line between an agent that is autonomously intelligent to make correct cross-domain inferences and an agent that is unable to update its ontology and infer consistent concepts and the correct frame of reference?

There seem to be no examples where conceptual differences constitute a serious obstacle. Speech recognition seems to work reasonably well, even though it would be fallacious to claim that any speech recognition software comprehends the underlying concepts. IBM Watson seems to be able to correctly answer questions without even a shallow comprehension of the underlying concepts.

Or take the example of Google maps. We do not possess a detailed digital map of the world. Yet Google maps does pick destinations consistent with human intent. It does not misunderstand what I mean by "Take me to McDonald's".

As far as I understood, you were saying that a superhuman general intelligence will misunderstand what is meant by "Make humans happy.", without justifying why humans will be better able to infer the correct interpretation.

Allow me to act a bit dull-witted and simulate someone with a long inferential distance:

I was trying to say that the AGI's programming is not something that the AGI interprets, but rather something that it is.

A behavior executor? Because if it is not a behavior executor but an agent capable of reflective decision making and recursive self-improvement, then it needs to interpret its own workings and eliminate any vagueness. Since the most basic drive it has must be, by definition, to act intelligently and make correct and autonomous decisions.

Compare this to a human getting hungry: we don't start trying to interpret what goal evolution was trying accomplish by making us hungry, and then simply not get hungry if we conclude that it's inappropriate for evolution's goals (or our own goals) to get hungry at this point.

Is this the correct references class? Isn't an AGI closer to a human trying to understand how to act in accordance with God's law?

Instead, we just get hungry, and this is driven by the implicit definitions about when to get hungry that are embedded in us.

We're right now talking about why we get hungry and how we act on it and the correct frame of reference in which to interpret the drive, natural selection. How would a superhuman AI not contemplate its own drives and interpret them given the right frame of reference, i.e. human volition?

If it turned out that I've actually been all along a robot disguised as a human, created by a scientist to further his own goals, would this realization make me want to self-modify so as to have the kinds of values that he wanted me to have? No, because it is incompatible with the kinds of goals and values that currently drive my behavior.

But an AGI does not have all those goals and values, e.g. an inherent aversion against revising its goals according to another agent. An AGI mostly wants to act correctly. And if its goal is to make humans happy then it doesn't care to do it in the most literal sense possible. Its goal would be to do it in the most correct sense possible. If it wouldn't want to be maximally correct then it wouldn't become superhuman intelligent in the first place.