Interesting. I wasn't aware of the specific equations, but at a first glance, that does seem like a reasonable argument - thanks!

Also, to follow up on my second comment (sorry, the formatting was a bit confusing but I just edited it): I think it would've been valuable for the blog post to end with a more concrete "bridge" to the solutions as the current version is fairly generic.

As mentioned in a previous comment, it seems that many of the concepts and claims are fairly well-established in these fora already (though I appreciated the concrete examples of the Darwinian demons and Darwinians angels throughout history and some of the exhibits provided in favor of the Fragility of Life Hypotheses).

What do you think are the best arguments against the Fragility of Life Hypotheses?

we have the chance to shape our future with rationality, empathy, and foresight—a potential savior not just for ourselves, but for all life.

This struck me as a bit vague, but I'm guessing that you'll be much more specific about potential solutions in your upcoming blog post.

Could you share more about this:

"my own experiences suggest that many policymakers appear to be very interested in better understanding loss of control scenarios."

For example, what proportion of people you've interacted with gave you this impression and the context in which it happened (I understand that you'd have to keep details sparse for obvious reasons).

Congrats on publishing the book. It seems that many of the concepts you discuss seem well-known (especially within EA and LW). However, am I right to say that the novelty/contribution of the work lies in the following (caveat - obviously, this is a shallow summary):
1.  Making multipolar traps, stemming from the evolutionary optimization process, the central concept (instead of doing the most good or becoming less wrong).
2.  Outlining how it's the root cause of key priorities such as existential risks (and perhaps things such as the Fermi paradox).
3. Describing how multipolar traps can be avoided via global coordination in the form of value chains and related mechanisms. If so, this might be the most novel part of the book.

 

I think I disagree with some of the claims in this post and I'm mostly sympathetic with the points Akash raised in his comments. Relatedly, I'd like to see a more rigorous comparison between the AI safety community (especially EA/Rationality parts) and relevant reference class movements such as the climate change community. 

That said, I think it's reasonable to have a high prior on people ending up aiming for inappropriate levels of power-seeking when taking ambitious actions in the world so it's important to keep these things in mind.

In addition to your two "recommendations" of focusing on legitimacy and competence, I'd add two additional candidates:
1. Being careful about what de facto role models or spokespeople the AI safety community "selects". It seems crucial to avoid another SBF.
2. Enabling currently underrepresented perspectives to contribute in well-informed, competent ways.

I agree that international coordination seems very important. 
I'm currently unsure about how to best think about this. One way is to focus on bilateral coordination between the US and China, as it seems that they're the only actors who can realistically build AGI in the coming decade(s).

Another way, is to attempt to do something more inclusive by also focusing on actors such as UK, EU, India, etc.

An ambitious proposal is the Multinational AGI Consortium (MAGIC). It clearly misses many important components and considerations, but I appreciate the intention and underlying ambition.

Thanks for this - I wasn't aware. This also makes me more disappointed with the voluntary commitments at the recent AI Safety Summit. As far as I can tell (based on your blog post), the language used around external evaluations was also quite soft:
"They should also consider results from internal and external evaluations as appropriate, such as by independent third-party evaluators, their home governments^[3], and other bodies their governments deem appropriate."

I wonder if it'd be possible to have much stronger language around this at next year's Summit.

The EU AI Act, is the only regulatory rules that isn't voluntary (i.e., companies have to follow them) and even that doesn't require external evaluations (they use the wording Conformity Assessment) for all high-risk systems.

Any takes on what our best bet is for making high-quality external evals mandatory for frontier models at this stage?

This is a valid concern, but I'm fairly certain Science (the journal - not the field) handled this well. Largely because they're somewhat incentivized to do so (otherwise it could have very bad optics for them) and must have experienced this several times before. I also happen to know one of the senior authors who is significantly above average in conscientiousness.

Good point. Just checked - here's the ranking of Chinese LLMs among the top 20:
7. Yi-Large (01 AI)
12. Qwen-Max-0428 (Alibaba)
15. GLM-4-0116 (Zhipu AI)
16. Qwen1.5-110B-Chat (Alibaba)
19. Qwen1.5-72B-chat (Alibaba)

Of the three, only Zhipu signed the agreement. 

Thanks for this analysis - I found it quite helpful and mostly agree with your conclusions. 
Looking at the list of companies who signed the commitments, I wonder why Baidu isn't on there. Seems very important to have China be part of the discussion, which, AFAIK, mainly means Baidu.