It occurs to me that we can view this proposal through the "acausal trade" lens, instead of the "indirect normativity" lens, which might give another set of useful intuitions. What Paul is proposing can be seen as creating an AGI that can exert causal control in our world but cares only about a very specific world / platonic computation defined by H and T, while the inhabitants of that world (simulated humans and their descendants) care a lot about our world but has no direct influence over it. The hoped for outcome is for the two parties to do a trade: the AGI turns our world into a utopia in return for the inhabitants of the HT World satisfying its preferences (i.e., having the computation return a high utility value).
From this perspective, Paul's proposal can also be seen as an instance of what I called "Instrumentally Friendly AI" (on the decision theory list):
Previous discussions about Rolf Nelson's AI Deterrence idea (or in Nesov's more general terms, UFAIs trading with FAIs across possible worlds) seem to assume that even at best we (or humane values) will get only a small share of any world controlled by a UFAI. But what if we can do much better (i.e., get almost the entire universe), by careful choice of the decision algorithm and utility function of the UFAI? In that case the UFAI might be worthy of the name "Instrumentally Friendly AI" (IFAI), and it might be something we'd want to deliberately build.
A recent post at my blog may be interesting to LW. It is a high-level discussion of what precisely defined value extrapolation might look like. I mostly wrote the essay while a visitor at FHI.
The basic idea is that we can define extrapolated values by just taking an emulation of a human, putting it in a hypothetical environment with access to powerful resources, and then adopting whatever values it eventually decides on. You might want some philosophical insight before launching into such a definition, but since we are currently laboring under the threat of catastrophe, it seems that there is virtue in spending our effort on avoiding death and delegating whatever philosophical work we can to someone on a more relaxed schedule.
You wouldn't want to run an AI with the values I lay out, but at least it is pinned down precisely. We can articulate objections relatively concretely, and hopefully begin to understand/address the difficulties.
(Posted at the request of cousin_it.)