I disagree; reading Paul's description made it clear to me how superficial it is to want to solve a problem by creating an army of uploads to do it for you. You may as well just try to solve the problem here and now, rather than hoping to outsource it to a bunch of nonexistent human-simulations running on nonexistent hardware. The only reason to consider such a baroque way of solving a problem is if you expect to be very pressed for time and yet to also have access to superdupercomputing power. You know, the world is hurtling towards singularity, no-one has crossed the finish line but many people are getting close, your FAI research organization manages to get a hold of a few petaflops on which to run a truncated AIXI problem-solver... and now you can finally go dig up that scrap of paper on which your team wrote down, years before, the perfectly optimal wish: "I want you, FAI-precursor, to do what the ethically stabilized members of our team would do, if they had hundreds of years to think about it, and if they...", etcetera.

It's a logically possible scenario, but is it remotely likely? This absolutely should not be the paradigm for a successful implementation of FAI or CEV. It's just a wacky contingency that you might want to spend a little time thinking about. The plan should be that un-uploaded people will figure out what to do. They will surely make intensive use of computers, and there may be some big final calculation in which the schematics of human genetic, neural and cultural architecture are the inputs to a reflective optimization process; but you shouldn't imagine that, like some bunch of Greg Egan characters, the researchers are going to successfully upload themselves and then figure out the logistics and the mathematics of a successful CEV process. It's like deciding to fix global warming by building a city on the moon that will be devoted to the task of solving global warming.

(Note that the hypothetical process probably doesn't even output a goal specification, it just outputs a number, which the AI tries to control.)

One potential worry is that the human subject must be over some minimal threshold of intelligence for this scheme to work. A village fool would fail. How do I convince myself that the threshold is below the "reasonably intelligent human" level?

The hope is something like: "We can reason about the outputs of this process, so an AI as smart as us can reason about the outputs of this process (perhaps by realizing it can watch or ask us to learn about the behavior of the definition)." The bar the AI has to meet, is to realize basically what is going on the definition. This assumes of course not only that the process actually works, but that it works for the reasons we believe it works.

I have doubts about this, and it seems generally important to think about whether particular models of UDT could make these inferences. The sketchy part seems to be the AI looking out at the world, and drawing mathematical inferences from the assumption that it's environment is a draw from a universal distribution. There are two ways you could imagine this going wrong:

1. Should / will the AI assume it's environment is universally distributed? It seems like we'll need something like this, so that the AI even has an understanding of how to build infrastructure, but when U is complicated it isn't clear. 1b. Can the AI even reason about where it's utility function came from, in order to draw inferences about it's mathematical properties? (this seems likely, given 1)
2. If the AI actually assumes it's environment is universally distributed, then it is living in a simulation, and maybe the simulator has some perverse motives and so the AI should draw very different conclusions from it's environment.

I expect UDT properly formulated avoids both issues, and if it doesn't we are going to have to take these things on anyway. But it needs some thought.

The "yield control" AGI seems problematic to define, much less to imagine existing. Do you think it is plausible?

This is certainly the line of thought that led me here, however.

I do give a (somewhat) concise overview, in the section headed 'The Proposal.'

The 100 years example is not quite right, in that in the real example we put you in an environment with unlimited computational power. One of the first things you are likely to do is create an extremely pleasant environment for yourself to work in (another is to create a community to work alongside you, either out of emulations of yourself, emulations of others, or reconstructed from simulations of worlds like Earth), while you figure out what should be done.

That said, there are other ways that your values might change through this process. For example, one of the first things you would hypothetically realize, if you ended up in an environment with some apparently infinitely powerful computers, is that you are in a hypothetical situation. I don't know about you, but if I discovered I was in a clearly hypothetical situation, my views about the moral relevance of people in hypotheticals would change (hypothetically).

We are trying to formally specify the input-output behavior of an idealized computer, running some simple program. The mathematical definition of a Turing machine with an input tape would suffice, as would a formal specification of a version of Python running with unlimited memory.

H is used to start off the process. H is then able to interact with a hypothetical unbounded computer, which may eventually run many (potentially quite exotic) simulations, among them of the sorts of minds humans self-modify into.

You kill the shell not when it is in an infinite loop, but when it takes more than a few seconds to run. We can set up such a safety net, allowing the human to run anything that takes (say) less than a million years to run, without risk of crashing. This is the sort of thing I was referring to by "some care."

Ultimately we do want the human to be able to run arbitrarily expensive subroutines, which prohibits using any heuristic of the form "stop this computation if it goes on for more than N steps."