It's got to confer some degree of dumbness avoidance.

Does it? On what grounds?

In any case, MIRI has already conceded that superintelligent AIs won't misbehave through stupidity. They maintain the problem is motivation ... the Genie KNOWS but doesn't CARE.

That's putting an alien intelligence in human terms; the very phrasing inappropriately anthropomorphizes the genie.

We probably won't go anywhere without an example.

Market economics ("capitalism") is an intelligence system which is very similar to the intelligence system Richard is proposing. Very, very similar; it's composed entirely of independent nodes (seven billion of them) which each provide their own set of constraints, and promote or demote information as it passes through them based on those constraints. It's an alien intelligence which follows Richard's model which we are very familiar with. Does the market "know" anything? Does it even make sense to suggest that market economics -could- care?

Does the market always arrive at the correct conclusions? Does it even consistently avoid stupid conclusions?

How difficult is it to program the market to behave in specific ways?

Is the market "friendly"?

Does it make sense to say that the market is "stupid"? Does the concept "stupid" -mean- anything when talking about the market?

An elementary error. The constraints in question are referred to in the literature as "weak" constraints (and I believe I used that qualifier in the paper: I almost always do). Weak constraints never need to be ALL satisfied at once. No AI could ever be designed that way, and no-one ever suggested that it would. See the reference to McClelland, J.L., Rumelhart, D.E. & Hinton, G.E. (1986) in the paper: that gives a pretty good explanation of weak constraints.

I understand the concept.

How exactly do you propose that the AI "weighs contextual constraints incorrectly" when the process of weighing constraints requires most of the constraints involved (probably thousands of them) to all suffer a simultaneous, INDEPENDENT 'failure' for this to occur?

I'd hazard a guess that, for any given position, less than 70% of humans will agree without reservation. The issue isn't that thousands of failures occur. The issue is that thousands of failures -always- occur.

Assuming this isn't more of the same, what you are saying here is isomorphic to the statement that somehow, a neural net might figure out the correct weighting for all the connections so that it produces the correctly trained output for a given input. That problem was solved in so many different NN systems that most NN people, these days, would consider your statement puzzling.

The problem is solved only for well-understood (and very limited) problem domains with comprehensive training sets.

A trivial variant of your second failure mode. The AI is calculating the constraints correctly, according to you, but at the same time you suggest that it has somehow NOT included any of the constraints that relate to the ethics of forced sterilization, etc. etc. You offer no explanation of why all of those constraints were not counted by your proposed AI, you just state that they weren't.

They were counted. They are, however, weak constraints. The constraints which required human extinction outweighed them, as they do for countless human beings. Fortunately for us in this imagined scenario, the constraints against killing people counted for more.

This is identical to your third failure mode, but here you produce a different list of constraints that were ignored. Again, with no explanation of why a massive collection of constraints suddenly disappeared.

Again, they weren't ignored. They are, as you say, weak constraints. Other constraints overrode them.

Another insult, and putting words into my mouth, and showing no understanding of what a weak constraint system actually is.

The issue here isn't my lack of understanding. The issue here is that you are implicitly privileging some constraints over others without any justification.

Every single conclusion I reached here is one that humans - including very intelligence humans - have reached. By dismissing them as possible conclusions an AI could reach, you're implicitly rejecting every argument pushed for each of these positions without first considering them. The "weak constraints" prevent them.

I didn't choose -wrong- conclusions, you see, I just chose -unpopular- conclusions, conclusions I knew you'd find objectionable. You should have noticed that; you didn't, because you were too concerned with proving that AI wouldn't do them. You were too concerned with your destination, and didn't pay any attention to your travel route.

If doing nothing is the correct conclusion, your AI should do nothing. If human extinction is the correct conclusion, your AI should choose human extinction. If sterilizing people with unhealthy genes is the correct conclusion, your AI should sterilize people with unhealthy genes (you didn't notice that humans didn't necessarily go extinct in that scenario). If rewriting minds is the correct conclusion, your AI should rewrite minds.

And if your constraints prevent the AI from undertaking the correct conclusion?

Then your constraints have made your AI stupid, for some value of "stupid".

The issue, of course, is that you have decided that you know better what is or is not the correct conclusion than an intelligence you are supposedly creating to know things better than you.

And that sums up the issue.