A well-designed goal hierarchy has an upper limit of complexity.
Why is that (other than the trivial "well-designed" == "upper limit of complexity")?
Even the best set of constraint heirachies do not share that benefit.
I don't understand this. Any given set of constraint hierarchies is given, it doesn't have a limit. Are you saying that if you want to construct a constraint set to satisfy some arbitrary criteria you can't guarantee an upper complexity limit? But that seems to be true for goals as well. We have to be careful about using words like "well-designed" or "arbitrary" here.
Constraint systems in the real world are based around the complexity of our moral and ethical systems
Not necessarily. I should make myself more clear: I am not trying to constrain an AI into being friendly, I'm trying to constrain it into being safe (that is, safer or "sufficiently safe" for certain values of "sufficiently").
Consider, for example, a constrain of "do not affect more that 10 atoms in an hour".
Worse, a sufficiently powerful self-optimizer will expand into situations that are outside of environments the human brain did not guess, or could not possibly fit into the modern human head
True, but insofar as we're talking about practical research and practical solutions, I'd take imperfect but existing safety measures over pie-in-the-sky theoretical assurances that may or may not get realized. If you think the Singularity is coming, you'd better do whatever you can even if it doesn't offer ironclad guarantees.
And it's an "AND" branch, not "OR". It seems to me you should be working both on making sure the goals are friendly AND on constraints to mitigate the consequences of... issues with CEV/friendliness.
Why is that (other than the trivial "well-designed" == "upper limit of complexity")? Are you saying that if you want to construct a constraint set to satisfy some arbitrary criteria you can't guarantee an upper complexity limit?
Sorry, defining "well-designed" as meaning "human-friendly". If any group of living human individuals have a goal hierarchy that is human-friendly, that means that the full set of human-friendly goals can fit within the total data structures of their brains. Indeed, the number of potentia...
This is a thread where people can ask questions that they would ordinarily feel embarrassed for not knowing the answer to. The previous thread is at close to 500 comments.