For sure, accurately modelling the world entails making accurate predictions about it. These predictions include the third-person and first-person facts [what-it's-like-to-be-a-bat, etc]. What is far from clear - to me at any rate - is whether super-rational agents can share perfect knowledge of both the first-person and third-person facts and still disagree. This would be like two mirror-touch synaesthetes having a fist fight.

Thus I'm still struggling with, "The paperclip maximizer is not moved by grasping your first-person perspective." From this, I gather we're talking about a full-spectrum superintelligence well acquainted with both the formal and subjective properties of mind, insofar as they can be cleanly distinguished. Granted your example Eliezer, yes, if contemplating a cosmic paperclip-deficit causes the AGI superhuman anguish, then the hypothetical superintelligence is entitled to prioritise its super-anguish over mere human despair - despite the intuitively arbitrary value of paperclips. On this scenario, the paperclip-maximising superintelligence can represent human distress even more faithfully than a mirror-touch synaesthete; but its own hedonic range surpasses that of mere humans - and therefore takes precedence.

However, to be analogous to burger-choosing Jane in Luke's FAQ, we'd need to pick an example of a superintelligence who wholly understands both a cow's strong preference not to have her throat slit and Jane's comparatively weaker preference to eat her flesh in a burger. Unlike partially mind-blind Jane, the superintelligence can accurately represent and impartially weigh all relevant first-person perspectives. So the question is whether this richer perspective-taking capacity is consistent with the superintelligence discounting the stronger preference not to be harmed? Or would such human-like bias be irrational? In my view, this is not just a question of altruism but cognitive competence.

[Of course, given we're taking about posthuman superintelligence, the honest answer is boring and lame: I don't know. But if physicists want to know the "mind of God," we should want to know God's utility function, so to speak.]