Sorry, I may have been confused about what you were trying to say because you were responding to someone else, and I hadn't come across the 'steelman' term before.

I withdraw 'parody' (sorry!) but ... it isn't quite what the logical structure of the paper was supposed to be.

It feels like you steelmanned it onto some other railroad track, so to speak.

I will take them one at a time:

Failure Mode I: The AI doesn't do anything useful, because there's no way of satisfying every contextual constraint.

An elementary error. The constraints in question are referred to in the literature as "weak" constraints (and I believe I used that qualifier in the paper: I almost always do). Weak constraints never need to be ALL satisfied at once. No AI could ever be designed that way, and no-one ever suggested that it would. See the reference to McClelland, J.L., Rumelhart, D.E. & Hinton, G.E. (1986) in the paper: that gives a pretty good explanation of weak constraints.

Predicting your response: "That's not what I meant."

That's an insult. But I will overlook it, since I know it is just your style.

Failure Mode II: The AI weighs contextual constraints incorrectly and sterilizes all humans to satisfy the sort of person who believes in Voluntary Human Extinction.

How exactly do you propose that the AI "weighs contextual constraints incorrectly" when the process of weighing constraints requires most of the constraints involved (probably thousands of them) to all suffer a simultaneous, INDEPENDENT 'failure' for this to occur?

That is implicit in the way that weak constraint systems are built. Perhaps you are not familiar with the details.

Predicting your response: "It would (somehow) figure out the correct weighting for all the contextual constraints."

Assuming this isn't more of the same, what you are saying here is isomorphic to the statement that somehow, a neural net might figure out the correct weighting for all the connections so that it produces the correctly trained output for a given input. That problem was solved in so many different NN systems that most NN people, these days, would consider your statement puzzling.

Failure Mode III: The AI weighs contextual constraints correctly (for a given value of "correctly") and sterilizes everybody of below-average intelligence or any genetic abnormalities that could impose costs on offspring, and in the process, sterilizes all humans.

A trivial variant of your second failure mode. The AI is calculating the constraints correctly, according to you, but at the same time you suggest that it has somehow NOT included any of the constraints that relate to the ethics of forced sterilization, etc. etc. You offer no explanation of why all of those constraints were not counted by your proposed AI, you just state that they weren't.

Predicting your response: "It wouldn't do something so dumb."

Yet another insult. This is getting a little tiresome, but I will carry on.

Failure Mode IV: The AI weighs contextual constraints correctly and puts all people of minority ethical positions into mind-rewriting machines so that there's no disagreement anymore.

This is identical to your third failure mode, but here you produce a different list of constraints that were ignored. Again, with no explanation of why a massive collection of constraints suddenly disappeared.

Predicting your response: "It wouldn't do something so dumb."

No comment.

We could keep going, but the issue is that so far, you've defined -any- failure mode as "dumb"ness, and have argued that the AI wouldn't do anything so "dumb", because you've already defined that it is superintelligent.

This is a bizarre statement, since I have said no such thing. Would you mind including citations, from now on, when you say that I "said" something? And please try not to paraphrase, because it takes time to correct the distortions in your paraphrases.

I don't think you know what intelligence -is-. Intelligence does not confer immunity to "dumb" behaviors.

Another insult, and putting words into my mouth, and showing no understanding of what a weak constraint system actually is.

I have explicitly addressed this point on many occasions. My paper had nothing in it that was specific to any failure mode.

The suggestion is that the entire class of failure modes suggested by Yudkowsky et al. has a common feature: they all rely on the AI being incapable of using a massive array of contextual constraints when evaluating plans.

By simply proposing an AI in which such massive constraint deployment is the norm, the ball is now in the other court: it is up to Yudkowsky et al. to come up with ANY kind of failure mode that could get through.

The scenarios I attacked in the paper have the common feature that they have been predicated on such a simplistic type of AI that they were bound to fail. They had failure built into them.

As soon as everyone moves on from those "dumb" superintelligences and starts to discuss the possible failure modes that could occur in a superintelligence that makes maximum use of constraints, we can start to talk about possible AI dangers. I'm ready to do that. Just waiting for it to happen, is all.

That is easy:

• Why would the humans have "not heard the plan yet"? It is a no-brainer part of this AI's design that part of the motivation engine (the goals) will be a goal that says "Check with the humans first." The premise in the paper is that we are discussing an AI that was designed as best we could, BUT it then went maverick anyway: it makes no sense for us to switch, now, to talk about an AI that was actually built without that most elementary of safety precautions!

• Quite independently, the AI can use its contextual understanding of the situation. Any intelligent system with such a poor understanding of the context and implications of its plans that it just goes ahead with the first plan off the stack, without thinking about implications, is an intelligent system that will walk out in front of a bus just because it wants to get to the other side of the road. In the case in question you are imagining an AI that would be capable of executing a plan to put all humans into bottles, without thinking for one moment to mention to anybody that it was considering this plan? That makes sense in any version of the real world. Such an AI is an implausible hypothetical.

Let me see if I can deal with the "no true scotsman" line of attack.

The way that that fallacy might apply to what I wrote would be, I think, something like this:

• MIRI says that a superintelligence might unpack a goal statement like "maximize human happiness" by perpetrating a Maverick Nanny attack on humankind, but Loosemore says that no TRUE superintelligence would do such a thing, because it would be superintelligent enough to realize that this was a 'mistake' (in some sense).

This would be a No True Scotsman fallacy, because the term "superintelligence" has been, in effect, redefined by me to mean "something smart enough not to do that".

Now, my take on the NTS idea is that it cannot be used if there are substantive grounds for saying that there are two categories involved, rather than a real category and a fake category that is (for some unexplained reason) exceptional.

Example: Person A claims that a sea-slug caused the swimmer's leg to be bitten off, but Person B argues that no "true" sea-slug would have done this. In this example, Person B is not using a No True Scotsman argument, because there are darned good reasons for supposing that sea-slugs cannot bite off the legs of swimmers.

So it all comes down to whether someone accused of NTS is inventing a ficticious category distinction ("true" versus "non-true" Scotsman) solely for the purposes of supporting their argument.

In my case, what I have argued is right up there with the sea-slug argument. What I have said, in effect, is that if we sit down and carefully think about the type of "superintelligence" that MIRI et al. put into their scenarios, and if we explore all the implications of what that hypothetical AI would have to be like, we quickly discover some glaring inconsistencies in their scenarios. The sea-slug, in effect, is supposed to have bitten through bone with a mouth made of mucous. And the sea-slug is so small it could not wrap itself around the swimmer's leg. Thinking through the whole sea-slug scenario leads us into a mass of evidence indicating that the proposed scenario is nuts. Similarly, thinking through the implications of an AI that is so completely unable to handle context, that it can live with Grade A contradictions at the heart of its reasoning, leads us to a mass of unbelievable inconsistencies in the 'intelligence' of this supposed superintelligence.

So, where the discussion needs to be, in respect of the paper, is in the exact details of why the proposed SI might not be a meaningful hypothetical. It all comes down to a meticulous dissection of the mechanisms involved.

To conclude: sorry if I seemed to come down a little heavy on you in my first response. I wasn't upset, it was just that the NTS critique had occurred before. In some of those previous cases the NTS attack was accompanied by language that strongly implied that I had not just committed an NTS fallacy, but that I was such an idiot that my idiocy was grounds for recommending to all not to even read the paper. ;-)

Hey, no problem. I was really just raising an issue with certain types of critique, which involve supposed fallacies that actually don't apply.

I am actually pressed for time right now, so I have to break off and come back to this when I can. Just wanted to clarify if I could.

Maybe I could try to reduce possible confusion here. The paper was written to address a category of "AI Risk" scenarios in which we are told:

"Even if the AI is programmed with goals that are ostensibly favorable to humankind, it could execute those goals in such a way that would lead to disaster".

Given that premise, it would be a bait-and-switch if I proposed a fix for this problem, and someone objected with "But you cannot ASSUME that the programmers would implement that fix!"

The whole point of the problem under consideration is that even if the engineers tried, they could not get the AI to stay true.

This is just a placeholder: I will try to reply to this properly later.

Meanwhile, I only want to add one little thing.

Don't forget that all of this analysis is supposed to be about situations in which we have, so to speak "done our best" with the AI design. That is sort of built into the premise. If there is a no-brainer change we can make to the design of the AI, to guard against some failure mode, then is assumed that this has been done.

The reason for that is that the basic premise of these scenarios is "We did our best to make the thing friendly, but in spite of all that effort, it went off the rails."

For that reason, I am not really making arguments about the characteristics of a "generic" AI.

Understood, and the bottom line is that the distinction between "terminal" and "instrumental" goals is actually pretty artificial, so if the problem with "maximize friendliness" is supposed to apply ONLY if it is terminal, it is a trivial fix to rewrite the actual terminal goals to make that one become instrumental.

But there is a bigger question lurking in the background, which is the flip side of what I just said: it really isn't necessary to restrict the terminal goals, if you are sensitive to the power of constraints to keep a motivation system true. Notice one fascinating thing here: the power of constraint is basically the justification for why instrumental goals should be revisable under evidence of misbehavior .... it is the context mismatch that drives that process. Why is this fascinating? Because the power of constraints (aka context mismatch) is routinely acknowledged by MIRI here, but flatly ignored or denied for the terminal goals.

It's just a mess. Their theoretical ideas are just shoot-from-the-hip, plus some math added on top to make it look like some legit science.

That isn't even remotely what the paper said. It's a parody.