I'm getting lost in my own argument.

If Michael was responding to the problem that human preference systems can't be unambiguously extended into new environments, then my chronologically first response applies, but needs more thought; and I'm embarrassed that I didn't anticipate that particular response.

If he was responding to the problem that human preferences as described by their actions, and as described by their beliefs, are not the same, then my second response applies.