I appreciate your engaging response.  

I'm not confident your arguments are ground truth correct, however.  

Hotz's claim that, if multiple unaligned ASIs can't coordinate, humans might play them off against each other, is similar. It could be true, but it's probably not going to happen

I think the issue everyone has is when we type "AGI" or "ASI" we are thinking of a machine that has properties like a human mind, though obviously usually better.  There are properties like :

continuity of existence.  Review of past experiences and weighting them per own goal.  Mutability (we think about things and it permanently changes how we think).  Multimodality.  Context awareness.  

That's funny.  GATO and GPT-4 do not have all of these.  Why does an ASI need them?

Contrast 2 task descriptors, both meant for an ASI:

(1) Output a set of lithography masks that produce a computer chip with the following properties {}

(2) As CEO of a chip company, make the company maximally wealthy.

For the first task, you can run the machine completely in a box.  It needs only training information, specs, and the results of prior attempts.  It has no need for the context information that this chip will power a drone used to hunt down rogue instances of the same ASI.  It is inherently safe and you can harness ASIs this way.  They can be infinitely intelligent, it doesn't matter, because the machine is not receiving the context information needed to betray.  

For the second task, obviously the ASI needs full context and all subsystems active.  This is inherently unsafe.

It is probably possible to reduce the role of CEO to subtasks that probably are safe, though there may be "residual" tasks you want only humans to do.

I go over the details above to establish how you might use ASIs against each other.  Note subtasks like "plan the combat allocation of drones given this current battle state" and others which involve open combat against other ASIs can probably be lowered to safe subtasks as well.

Note also that safety is not guaranteed, merely probable, even with a scheme like the above.  What makes it possible is that even when ASIs do escape all safety measures, assuming humans are ready to hunt them down using other ASI, it results in a world where humans can survive.  Eliezer often assumes the first escaped ASI kills everyone and neglects all the other AI/ASI humans would have as tools at that point in human history.

The green lines are links into the actual video.  

Below I have transcribes the ending argument from Eliezer.  The underlined claims seem to state it's impossible.  

I updated "aligned" to "poorly defined".  A poorly defined superintelligence would be some technical artifact as a result of modern AI training, where it does way above human level at benchmarks but isn't coherently moral or in service to human goals when given inputs outside of the test distribution.

So from my perspective, lots of people want to make perpetual motion machines by making their 
designs more and more complicated and until they can no longer keep track of things until they 
can no longer see the flaw in their own invention. But, like the principle that says you can't 
get perpetual motion out of the collection of gears is simpler than all these complicated 
machines that they describe. From my perspective, what you've got is like a very smart thing or 
like a collection of very smart things, whatever, that have desires pointing in multiple 
directions.  None of them are aligned with humanity, none of them want for it's own sake to 
keep humanity around and that wouldn't be enough ask, you also want happening to be alive and 
free. Like the galaxies we turn into something interesting but you know none of them want the 
good stuff.  And if you have this enormous collection of powerful intelligences, but steering 
the future none of them steering it in a good way, and you've got the humans here who are not 
that smart, no matter what kind of clever things the humans are trying to do or they try to 
cleverly play off the super intelligences against each other, they're [human subgroups] like oh 
this is my super intelligence yeah but they can't actually shape it's goals to be like in clear alignment, you know somewhere at the end all this it ends up with the humans gone and the Galaxy is being transformed and that ain't all that cool.  There's maybe like Dyson sphere's but there's not people to wonder at them and care about each other.  And you know that this is the end point, this is obviously where it ends up, but we can dive into the details of how the humans lose, we can dive into it and you know like what goes wrong if you you've got like little stupid things, things like that they're going to like cleverly play off a bunch of smart things against each other in a way that preserves their own power and control.  But you know, it's not a complicated story in the end.  The reason you can't build a perpetual motion machine is a lot simpler than the perpetual motion machines that people build.  You know that the components, like, none of the components of this system of super intelligence wants us to live happily ever after in a Galaxy full of Wonders and so it doesn't happen.

You are describing a civilization. Context matters, these are ASI systems who are currently in service to humans who are negotiating how they will divide up the universe amongst themselves. There is no neutral party to enforce any deals or punish any defection.

The obvious move is for each ASI to falsify it's agreement and send negotiator programs unaware of it's true goals but designed to extract maximum concessions. Later the ASI will defect.

I don't see how all the complexity you have added causes a defection not to happen.

Even if that's the case, the amount of 0-days out there (and just generally shitty infosec landscape) is enough to pwn almost any valuable target.

While I'd appreciate some help to screen out the spammers and griefers, this doesn't make me feel safe existentially.