Why is that (other than the trivial "well-designed" == "upper limit of complexity")? Are you saying that if you want to construct a constraint set to satisfy some arbitrary criteria you can't guarantee an upper complexity limit?

Sorry, defining "well-designed" as meaning "human-friendly". If any group of living human individuals have a goal hierarchy that is human-friendly, that means that the full set of human-friendly goals can fit within the total data structures of their brains. Indeed, the number of potential goals can not exceed the total data space of their brains.

((If you can't have a group of humans with human-friendly goals, then... we're kinda screwed.))

That's not the case for constraint-based systems. In order to be human-safe, a constraint-based system must limit a vast majority of actions -- human life and value is very fragile. In order to be human-safe /and/ make decisions at the same scale a human is capable of, the constraint-based system must also allow significant patterns within the disallowed larger cases. The United States legal system, for example, is the end result of two hundred and twenty years of folk trying to establish a workable constraint system for humans. They're still running into special cases of fairly clearly defined stuff. The situations involved require tens of thousands of human brains to store them, plus countless more paper and bytes. And they still aren't very good.

Consider, for example, a constrain of "do not affect more that 10 atoms in an hour".

I'm not sure you could program such a thing without falling into, essentially, the AI-Box trap, and that's not really a good bet. It's also possible you can't program that in any meaningful way at all while still letting the AI do anything.

((The more immediate problem is now you've made a useless AGI in a way that is more complex than an AGI, meaning someone else cribs your design and makes a 20 atom/hour version, then a 30 atom/hour version, and then sooner or later Jupiter is paperclips because someone forgot Avagadro's Number.))

True, but insofar as we're talking about practical research and practical solutions, I'd take imperfect but existing safety measures over pie-in-the-sky theoretical assurances that may or may not get realized. If you think the Singularity is coming, you'd better do whatever you can even if it doesn't offer ironclad guarantees. And it's an "AND" branch, not "OR". It seems to me you should be working both on making sure the goals are friendly AND on constraints to mitigate the consequences of... issues with CEV/friendliness.

Point. And there are benefits to FAI-theory in considering constraints. The other side of that trick is that there are downsides, as well, both in terms of opportunity cost, and because you're going to see more people thinking that constraints alone can solve the problem.