I wish I could believe that.

To start off, I agree with pretty much all of it. It's unlikely that any of the main players actively want the world to end, and inasmuch as they'll bring that outcome about, it'll be by mistake. It's marginally more likely that some of them are risking the world in the pursuit of personal power/status/ideology, and monstrously consider everyone's deaths an acceptable risk, but I'm not certain of that either.

That said, I agree that we could, in principle, all cooperate here. The thing often touted about as a counterargument is "but China". Except China doesn't want to die any more than the US, and there isn't, in principle, any reason the Chinese government can't be convinced of the seriousness of the danger. They would believe in the reality of an asteroid on a collision course with Earth if shown the evidence, and the AGI threat is no less real.

What we all can cooperate towards is to ban AGI development, and seek other, more controllable and less unilateral ways to create superintelligences. Human cognitive enhancement, genetic engineering, uploads, whatever.

What we can all do, together, is avert the omnicide. It's in ~no-one's interests.

What I don't believe is that we could cooperate on building an AGI that would bring about an utopia.

The term "AI Alignment" is a bit obfuscatory. The technical problem of AI Alignment would be better termed the control problem. An aligned AGI isn't necessarily an AGI that does the best for humanity; an aligned AGI is one that does precisely what its designer(s) intended for it to.

And the utopias of the majority of people, or companies, or governments, across history and across the world today, would be death or a hellscape for most other people. We would not enjoy life in a world in which North Korea, or a corporate sociopath, or Trump's US, or some bureaucracy, solved AGI alignment.

Again, I would like to empathize that "that means we need to outrace everyone else" is not the correct takeaway from this.

• Firstly, who are "we"? There is no person or a group of people Humanity can trust with doing it right.
  • No-one who is currently racing can be trusted. A US company successfully solving alignment isn't particularly more likely to result in a pro-humanity utopia, or an utopia for you, than the Chinese doing it first.
  • No-one who may start racing can be trusted. The US military nationalizing the project isn't going to result in a good outcome, either. And if we set things up such that the AI's value system is being decided as a matter of national policy, via a democratic process? Where people discuss it among themselves, with politicians chiming in, and maybe then vote on it, and then the result is interpreted by some bureaucratic process, and then someone types it in? Don't make me laugh. We'd be lucky if the AI that'd result from this just kills everyone and tiles the universe with paperwork, rather than trapping everyone in some inescapable Kafkaesque nightmare.
    • (To be clear, it's not because the people will want bad things. It's because our processes for eliciting and agglomerating their preferences – any and all processes in wide use – are an abomination.)
  • To do it right, whatever process builds the AGI would need to be actively, desperately trying not to leave its fingerprints on the future. But I trust pretty much no-one and no thing to actually do that, instead of hijacking the future for their values. ... Except myself and a few specific people, of course. But I rather doubt that you or the people of Indonesia or humanity as a whole would feel particularly enthusiastic about handing off that decision to me, yeah?
  • Same for any other candidature. There is nobody to whom Humanity as a whole can entrust its future; nobody to whom it should feel comfortable deferring that decision.
• Secondly, that'd be giving up too early. None of the above invalidates the core argument: even if we can't agree on an utopia, pretty much all of us would prefer to keep things as they are, keep incrementally improving our conditions and painstakingly negotiating compromises, to all of us just dying overnight. And the policy of "so we need to outrace the competitors before they build a hell" doesn't actually lead to your utopia, because you are not going to solve alignment in those conditions either. That policy leads to death. Which, again, ~nobody wants.

So I agree that we can all cooperate on this, that the current state of affairs is a mistake, and that we can negotiate for a better outcome. Ban AGI research internationally. Keep advancing using other technologies.

We will still need to reconcile our differences later on, of course. But it can be done incrementally, a steady pace of negotiation and power balancing and cultural mingling and sanity-raising. There are routes of intelligence enhancement that are more gradual, that let us preserve this sort of incrementalism and stability while still letting Humanity keep empowering itself. Gradual intelligence augmentation, via biotechnological or cyborgism-like or upload-based means. 

Humanity-as-a-whole can't entrust its future to any given part of itself. But it can still build a future for itself. There doesn't have to be a singular point in time at which we are deciding the whole shape of the future and are then unable to backtrack.

Bottom line: As things stand, anyone anywhere solving either AGI or AGI alignment is not, on expectation, going to lead to a good outcome for humanity. Our processes are too dysfunctional:

• We can't trust each other to let each other solve alignment in peace, and–
• – we can't survive if we let that distrust get to us and start racing each other, because then none of us solve alignment and we all die.

The best outcome we can all cooperate towards – and that is a good outcome that we can all cooperate towards – is to ban the accursed thing.

Big upvote for looking for routes to cooperation instead of either despairing or looking for reasons for conflict.

This all got a little long, so I'll put the biggest conclusion up front: I think we're in a good situation right now, in which the leading players pursuing AGI are probably not sadists, dense psychopaths, or zealots of mistaken ideologies. We'd probably like their utopias just fine. If we assume the competition to control aligned AGI will be much broader, we have more reason to be concerned.

One major crux of this post's claims is the intuition that there would be only minor variations in the "utopia" brought about by different actors with an aligned ASI. Intuitions/theories seem to vary widely on this point. OP didn't present much argument for that, so let me expand on it a little.

In sum, it's a question about human nature. Given unlimited power, will people use it to give people what they want, or will they enforce a world order most people hate?

This of course depends on the individuals and ideologies that wind up controlling that AGI.

It requires little benevolence to help people when it's trivially easy for the people in charge to do it.

This is one reason for optimism. It's based on the prediction that aligned AGI becomes aligned ASI and ASI can create a post-scarcity world. In a post-scarcity world, everyone can easily be given material resources that empower them to do whatever they want.

The pessimistic view is that some people or organizations don't have even the small bit of benevolence required to do good when it's trivially easy.

The thesis is that other motivations would outweigh their small amount of empathy/benevolence. This could be sadism; desire for revenge for perceived harms; or sincere belief in a mistaken worldview (e.g., religion or other prescriptive forms of ethics).

I think those possibilities are real, but we must also ask how those ideologies and emotional preferences would change over time.

Another reason for pessimism is not believing or not considering the post-scarcity hypothesis. The way corporations and individuals wield power in a world with scarcity does not inspire confidence. But the profit motive barely applies once you've "won that game". How would a corporation with access to unlimited production use that power? I think it would depend on the particular individuals in power, and their ideologies. And the power motive that's so destructive loses much of its emotional force once that individual has attained nearly unlimited power.

The creators of aligned AGI have won whatever game they've been playing. They have access to unlimited personal wealth and power. They and their loved ones are permanently safe from physical harms. No individual in history has ever been in that position.

I think the common saying "power corrupts" is quite mistaken, in an understandable way. The pursuit of power is what corrupts. It rewards unethical decisions, and provides pressure for the individual to see those decisions as ethical or virtuous. This corrupts their character. Every leader in history has had legitimate concerns about preserving their power. The individuals controlling AGI would not. If this is correct, the question is how corrupted they became while acquiring power, and whether they'll over time become more generous, as the legitimate reasons for selfishness disappear in reality, and perhaps in their emotional makeup as a result.

There's another important concern that sociopaths tend to acquire power in our current systems, while hiding their sociopathy. I think this is true, but we don't know how common it is. And we don't know that much about sociopathy. I think it probably exists on a spectrum, since it doesn't have a single source of genetic variance.

So, I hold out some hope that even most people on the sociopathy spectrum, or ideologically confused power structures would shift toward having the minimal benevolence-balance to provide a pretty awesome utopia. But I'd prefer to gamble on the utopias offered by Altman, Hassabis, or Amodio. This is an argument against an AI pause, but not a strong one.

You have a hypothesis that powerful people are amenable to reason. I see a lot of countervailing evidence.

Your strong conclusion 

The people currently taking actions that increase the probability that {the former is solved first} are not evil people trying to kill everyone, they're confused people who think that their actions are actually increasing the probability that {the latter is solved first}.

rests on a few over-simplifications associated with

Almost all of what the future contains is determined by which of the two following engineering problems is solved first:

• How to build a superintelligent AI (if solved first, everyone dies forever)
• How to build an aligned superintelligent AI (if solved first, everyone gets utopia)

For example:

• Say, I'm too old to expect aligned AI to give me eternal life (or aligned AI simply might not mean eternal life/bliss for me, for whichever reason; maybe as it's still better to start with newborns more efficiently made into bliss-enjoying automatons or whatever utopia entails), so for me individually, the intermediate years before superintelligence are the relevant ones, so I might rationally want to earn money by working on enriching myself, whatever the (un-)alignment impact of it
• Given the public goods nature of alignment, I might fear it's unlikely for us to cooperate, so we'll all freeride by working to enrich ourselves working on various things rather leaning towards building unaligned AI. With such a prior, it may be rational for any self-interested person - also absent confusion - to indeed freeride: 'hopefully I make a bit of money for an enjoyable few or many years, before unaligned AGI destroys us with near-certainty anyway'.
• Even if technical alignment was not too difficult, standard Moloch-type effects (again, all sorts of freeriding/power seeking) might mean chances of unaligned users of even otherwise 'aligned' technology are overwhelming, again meaning most value for most people lies in increasing their material welfare for the next few years to come, rather than 'wasting' their resources towards an futile alignment project.
• Assume we're natural humans, instead of perfectly patient beings (I know, weird assumption). So you're eating the chocolate and drinking the booze, even if you know in the longer-term future it's a net negative for you. You need not be confused about it, you need not even be strictly irrational from today's point of view (depending on the exact definition of the term): As you might genuinely care more about the "you" in the current and coming few days or years, than about that future self that simply doesn't yet feel too close to you - in other words, just the way it can be rational to care a bit more about yourself than about others, you might care a bit more about your 'current' self than about that "you" a few decades - or an eternity - ahead. So, the near-eternal utopia might mean a lot to you, but not infinitely much. It is easy to see that then, it may well remain rational for you to use your resources towards more mundane aims of increasing your material welfare for the few intermediate years to come - given that your own potential marginal contribution to P(Alignment first) is very small (<<1).

Hence, I'm afraid, when you take into account real-world complexities, it may well be rather perfectly rational for individuals to race on towards unaligned superintelligence; you'd require more altruism rather than only more enlightenment to improve the situation. In a messy world of 8 billions in capitalistic partly-zero-sum competition (or here even partly negative-sum competition), it simply isn't simple to get cooperation even if individuals were approximately rational and informed - even if, indeed, we are all in this together.

Nitpicking a particular topic of interest to me:

Power/money/being-the-head-of-OpenAI doesn't do anything post-singularity.

It obviously does?

I am very confused why people make claims in this genre. "When the Singularity happens, this (money, conflict, the problems I'm experiencing) won't be a problem anymore."

This mostly strikes me as magical, far-mode thinking. It's like people have an afterlife-shaped hole after losing religion. The specific, real reality in front of you won't magically suddenly change after an Intelligence Explosion and assuming we're alive in some coherent state. Money and power are very, very likely to still exist afterwards, just in a different state that makes sense as a transformation of the current world.

Power/money/being-the-head-of-OpenAI doesn't do anything post-singularity.

Any realistic practical 'utopia' will still have forms of money (fungible liquid socio-economic power), and some agents will have vastly more of it than others: like today, except only moreso.

I appreciate these views being stated clearly, and at once feel a positive feeling toward the author, and also am shaking my head No. As others have pointed out, the mistake theory here is confused.

I think it's not exactly wrong. The way in which it's right is this:

If people doing AGI research understood what we understand about the existential risk of AGI, most of them would stop, and AGI research would go much slower.

In other words, most people are amenable to reason on this point, in the sense that they'd respond to reasons to not do something that they've been convinced of. This is not without exception; some players, e.g. Larry Page (according to Elon Musk), want AGI to take the world from humanity.

The way in which the mistake theory is wrong is this:

Many people doing AGI research are not trying, and in many cases trying not, to understand what we understand about AGI risk.

So it's not just a mistake. It's a choice, that choice has motivations, and those motivations are in conflict with our motivations, insofar as they shelter themselves from reason.

And if you're not using your power/money to affect which of those two outcomes is more likely to happen than the other, then your power/money is completely useless. They won't be useful if we all die, and they won't be useful if we get utopia.

I disagree with this, because I think the following three things are true:

1. There is a finite amount of value in the accessible universe (or multiverse, or whatever).
2. Some people have unbounded "values", especially around positional goods like status among other humans.

A way I imagine this concretely playing out, conditional on intent alignment succeeding, is that very powerful post-human beings descended from people who controlled AI during the pivotal period playing very costly status games with each other, constructing the cosmic equivalent of the Bughatti Veyron or the Oman Royal Yacht Squadron, without being concerned with impartial value. I still expect them to provide for the "basic" needs of humanity, because it is so incredibly cheap, making it a utopia for people with bounded or modest goals, but e.g. preventing impartial hedonic utilitarians or people with many positional or nosy values from enacting their goals.

This depends on the people ultimately in charge of powerful AI systems to be philosophically unsophisticated, but most people are philosophically unsophisticated, and philosophical sophistication appears mostly uncorrelated with engineering or business success, so this doesn't seem like a bottleneck.

This view of course fails when single individuals become exceedingly powerful, in which case I don't have as strong a story. I'd be interested in what individual humans have historically done when they were strongly dominant over all forces around them.

It's encouraging to see more emphasis recently on the political and public-facing aspects of alignment. We are living in a far-better-than-worst-case world where people, including powerful ones, are open to being convinced. They just need to be taught - to have it explained to them intuitively.

It seems cached beliefs produced by works like you get about five words have led to a passive, unspoken attitude among many informed people that attempting to explain anything complicated is futile. It isn't futile. It's just difficult.