As Eliezer Yudkowsky thinks, I am afraid that AGI will by default end humanity. I also agree that there is only a point in making an aligned AGI if that AGI is powerful enough to block all following AGIs (the pivotal act). If not, an unaligned AGI is sooner or later going to emerge, and that AGI is likely to win over aligned AGIs because it's more efficient. Evolutionary pressure works against us. I also agree that the chance we succeed in doing this, is small.

However, unlike Yudkowsky, I think there might be a non-vanishing chance of doing what he calls the pivotal act, and I would call AGI regulation, before the first AGI, instead of after. This regulation should be decided democratically and be robust, while doing as little economic damage as possible (this might still be a significant amount). The public will likely accept this regulation if it wholeheartedly believes that the alternative might be human extinction. A deal between the US and China could be sufficient. This deal would be achievable, as long as elites and the public in these countries believe that the alternative is likely to be extinction. Note as well that the result would likely be the same as after Yudkowsky's pivotal act, since an AGI might not be able to do anything more than only this act (since aligning just this act is already difficult enough).

The big problem here is not coordination, but communication. For this route to work, we need elites and regular people, at least roughly 50%, to wholeheartedly believe that AI could cause human extinction. Convincing people about this is hard. There are a few reasons for that. First, science fiction has made fun of it for decades, meaning there is a significant social penalty for anyone bringing up the idea in a serious context. Few things are more effective at making someone shut up than the fear of being ridiculous, and therefore information doesn't spread nearly as fast as it otherwise would. Second, there is no scientific consensus. The majority of AI academics (although not the AI safety ones) dismisses the idea (the scifi effect, conflicts of interest, and undue conservatism likely all play a role). And third, there are the biases that have been described by Yudkowsky and others before.

Basically all forms of AI xrisk communication have been tried, including newspaper articles, books, scary youtube videos, TED talks, and documentaries. There is no single communication method that is going to convince everyone in one go. But still, real progress can be made.

In the last year, the organization I have founded has piloted communication to a general audience in the Netherlands. With a self-funded team of only about 3 FTE, we have succeeded in getting four pieces in major newspapers published. This meant increasing total coverage of the topic by more than 25%. Also, after our intervention, a leading opinion maker, who was skeptical about the topic before, started to write about it (text in Dutch). I think that in a few years, we could really change the public opinion on the topic. In my model, achieving this in the US and China would give us a real shot at globally effective AGI regulation. And that could mean saving us.

If a small self-funded organization like ours can do this, imagine what ten or fifty large AI xrisk communication organizations, well-funded and staffed with people who are great at communication, active in the most impactful regions, could do.

What we need, are people who want to help. Not by staffing our org, we don't have problems attracting talent. Also not primarily by funding us (although we could productively spend extra money). But mostly by starting great AI xrisk communication initiatives and organizations themselves, much better ones than ours. Because if we all work on this in earnest, we have a real chance.