Also worth noting is Steve Hsu's recent discussion of his meetings with China VC, government, researchers etc. reporting from on the ground in Shanghai and Beijing etc: https://www.manifold1.com/episodes/letter-from-shanghai-reflections-on-china-in-2024-73/transcript (which parallels Dwarkesh Patel's recent trip to China).
Hsu (better known for other work) is a long-time China hawk and has been talking up the scientific & technological capabilities of the CCP for a long time, saying they were going to surpass the West any moment now, so I found this interesting when Hsu explains that:
-
the scientific culture of China is 'mafia' like (Hsu's term, not mine) and focused on legible easily-cited incremental research, and is against making any daring research leaps or controversial breakthroughs...
but is capable of extremely high quality world-class followup and large scientific investments given a clear objective target and government marching orders
-
there is no interest or investment in an AI arms race, in part because of a "quiet confidence" (ie. apathy/lying-flat) that if anything important happens, fast-follower China can just catch up a few years later and win the real race. They just aren't doing it. There is no Chinese Manhattan Project. There is no race. They aren't dumping the money into it, and other things, like chips and Taiwan and demographics, are the big concerns which have the focus from the top of the government, and no one is interested in sticking their necks out for wacky things like 'spending a billion dollars on a single training run' without explicit enthusiastic endorsement from the very top.
Let the crazy Americans with their fantasies of AGI in a few years race ahead and knock themselves out, and China will stroll along, and scoop up the results, and scale it all out cost-effectively and outcompete any Western AGI-related stuff (ie. be the BYD to the Tesla). The Westerners may make the history books, but the Chinese will make the huge bucks.
So, this raises an important question for the arms race people: if you believe it's OK to race, because even if your race winds up creating the very race you claimed you were trying to avoid, you are still going to beat China to AGI (which is highly plausible, inasmuch as it is easy to win a race when only one side is racing), and you have AGI a year (or two at the most) before China and you supposedly "win"... Then what?
- race to AGI and win
- trigger a bunch of other countries racing to their own AGI (now that they know it's doable, increasingly much about how to do it, can borrow/steal/imitate the first AGI, and have to do so "before it's too late")
- ???
- profit!
What does winning look like? What do you do next? How do you "bury the body"? You get AGI and you show it off publicly, Xi blows his stack as he realizes how badly he screwed up strategically and declares a national emergency and the CCP starts racing towards its own AGI in a year, and... then what? What do you do in this 1 year period, while you still enjoy AGI supremacy? You have millions of AGIs which can do... stuff. What is this stuff? Are you going to start massive weaponized hacking to subvert CCP AI programs as much as possible short of nuclear war? Lobby the UN to ban rival AGIs and approve US carrier group air strikes on the Chinese mainland? License it to the CCP to buy them off? Just... do nothing and enjoy 10%+ GDP growth for one year before the rival CCP AGIs all start getting deployed? Do you have any idea at all? If you don't, what is the point of 'winning the race'?
(This is a question the leaders of the Manhattan Project should have been asking themselves when it became obvious that there were no genuine rival projects in Japan or Germany, and the original "we have to beat Hitler to the bomb" rationale had become totally irrelevant and indeed, an outright propaganda lie. The US got The Bomb, immediately ensuring that everyone else would be interested in getting the bomb, particularly the USSR, in the foreseeable future... and then what? Then what? "I'll ask the AGIs for an idea how to get us out of this mess" is an unserious response, and it is not a plan if all of the remaining viable plans the AGIs could implement are one of those previous plans which you are unwilling to execute - similar to how 'nuke Moscow before noon today' was a viable plan to maintain nuclear supremacy, but wasn't going to happen, and it would have been better to not put yourself in that position in the first place.)
EDIT: because people are accusing me of lying or being insane or maliciously distorting what Hsu said, let me excerpt the relevant Hsu parts, and highlight the key parts which I am condensing as 'there are large, conventional-style investments in AI and chipmaking, but there is no Chinese Manhattan Project, because of a government-level strategic assessment that a fast-follower strategy is adequate and ensures a long-term favorable outcome, focused primarily on chip manufacturing':
...While I was in Beijing, I also met with some top venture capitalists and technologists. I again can't say too much about it. I just want to say that there's quiet confidence throughout all, among all the people in China, whether it was academic scientists, technologists, investors, venture capitalists, business people, just quiet confidence that nothing the outside world, specifically the U. S., can do is really going to stop the rise of China.
And in particular, a lot of conversation was about AI and the chip war. And there's a sense of quiet confidence here that China's going to get the AI training done that it needs to do. It's not going to fall way behind in the race for AGI or ASI. There are government national level plans in place to build the data centers, to produce domestically the chips necessary to run those data centers, to power those data centers, and to stay abreast of developments in AI and also in frontier chip manufacturing.
Let's just say that there's quiet confidence here. That, you know, they may not fully catch up. They may not get their EUV machine for some number of years, but they're not really worried. And so, and many people have said to me that the very stupid Biden Jake Sullivan chip war against China has only helped Chinese companies. This is something I've discussed in other podcasts, when the U. S. cuts off access for Chinese companies to key products and technologies used in the semiconductor supply chain from the U. S. and say Dutch companies like ASML, Japanese companies as well. When the U. S. starts to threaten that, it only causes a coalescence of effort here in China. It creates a necessary coordination of effort here that then lets the Chinese supply chain ecosystem for semiconductors advance very rapidly.
And so it was, it was a stupid policy by the Biden administration. And it was also based on a miscalibrated estimate of how fast we were going to get to AGI. They thought, Oh, if we just, if we just kneecap the Chinese right now, since we're [sure?] AGI is right around the corner, this will let America get to super AGI and the Chinese will be behind and then they'll be screwed. And it doesn't look like it's playing out that way. Let's just put it that way.
I can't say much more about the details of what I learned on this trip.
But I think quiet confidence and a sense of inevitability in that sector, but across all sectors here. No, people here are just confident that like, oh, if there's some product that needs to be produced, batteries, photovoltaics, cars, robotics, factory automation, 5G, 6G, whatever it is, even leading edge CPUs, leading edge semiconductor nodes. There's just a quiet confidence that China's going to get there or is already there. And actually, in many of these things, they just don't feel like Westerners can compete. They just don't feel like it, if it's something that has to be made in a factory, it's eventually going to be made by Chinese companies, not by Western companies that the Westerners just can't compete with.
I mean, literally that, that may sound very jarring for you to hear that, but between two Chinese people who are technologists and know their stuff, it's a very common sentiment. You know, it's just a very common sentiment. Once China figures out how to produce it, [then] they can generally produce it much more efficiently than Western companies.
And it's not just a matter of labor costs. There is a labor cost advantage here, but it's much more than that. It's the power of the infrastructure here. It's the ability of the government and the companies to work together to be efficient. It's the willingness of the teams to work really hard here. Okay, I'll comment a little bit more about that when I talk about, say, SMIC and other stuff.
Further:
...They're not against big breakthroughs in PRC, esp. obvious ones. The bureaucracy bases promotions, raises, etc. on metrics like publications in top journals, citations, ... However there are very obvious wins that they will go after in a coordinated way - including AI, semiconductors, new energy tech, etc.
...There is coordinated planning btw govt and industry in PRC to stay at the frontier in AI/AGI/ASI. They are less susceptible to "visionaries" (ie grifters) so you'll find fewer doomers or singularitarians, etc. Certainly not in the top govt positions. The quiet confidence I mentioned extends to AI, not just semiconductors and other key technologies.
3
3
1Benjamin Todd reports back from "a two-week trip in China" on "Why a US AI 'Manhattan Project' could backfire: notes from conversations in China" (cf Dwarkesh), hitting very similar points about lack of funding/will despite considerable competence, and that:
So what might trigger a wake up? Most people said they didn’t know. But one suggestion was that the fastest way would be a high-profile US state-led AI project (especially if its explicit goal is US dominance…).
This means calls for a US "Manhattan Project" for AGI might easily be self-defeating. If maintaining a technological lead is your goal, better to stfu and hope the status quo persists as long as possible. (Or if you do go ahead, you need much stricter export restrictions.)
Steve Hsu clarified some things on my thread about this discussion: https://x.com/hsu_steve/status/1861970671527510378
"Clarifications:
1. The mafia tendencies (careerist groups working together out of self-interest and not to advance science itself) are present in the West as well these days. In fact the term was first used in this way by Italian academics.
2. They're not against big breakthroughs in PRC, esp. obvious ones. The bureaucracy bases promotions, raises, etc. on metrics like publications in top journals, cititations, ... However there are very obvious wins that they will go after in a coordinated way - including AI, semiconductors, new energy tech, etc.
3. I could be described as a China hawk in that I've been pointing to a US-China competition as unavoidable for over a decade. But I think I have more realistic views about what is happening in PRC than most China hawks. I also try to focus on simple descriptive analysis rather than getting distracted by normative midwit stuff.
4. There is coordinated planning btw govt and industry in PRC to stay at the frontier in AI/AGI/ASI. They are less susceptible to "visionaries" (ie grifters) so you'll find fewer doomers or singularitarians, etc. Certainly not in the top govt positions. The quiet confidence I mentioned extends to AI, not just semiconductors and other key technologies."
(All of which I consider to be consistent with my summary, if anyone is wondering, and thus, given that Hsu did not choose to object to any of the main points of my summary in his clarifications, are confirmation.)
Do you have any idea at all? If you don't, what is the point of 'winning the race'?
Maybe they have some idea but don't want to say it. In recently disclosed internal OpenAI emails, Greg Brockman and Ilya Sutskever said to Elon Musk:
Perhaps this - originally private email - is saying the quiet part. And now that it is released, the quiet part is out loud. To use terms from the turn based game of Civilisation, perhaps they would use AI to achieve a cultural, espionage, technological, influence, diplomatic, and military victory simultaneously? But why would they declare that beforehand? Declaring it would only invite opposition and competition.
At the very least, you can hack and spy and sabotage other AGI attempts.
To be specific, there are a few areas where, it seems to me, increased intelligence could lead to quick and leveraged benefits. Hacking, espionage, negotiation, finance, and marketing/propaganda. For example, what if you can capture a significant fraction of the world's trading income, attract a large portion of China's talent to turn coat and move to your country, and hack into a large part of an opposition's infrastructure.
If one or more of these tactics can work significantly, you buy time for other tactics to progress.
Jess Riedel has the most useful answer here to this question:
So, this raises an important question for the arms race people: if you believe it's OK to race, because even if your race winds up creating the very race you claimed you were trying to avoid, you are still going to beat China to AGI (which is highly plausible, inasmuch as it is easy to win a race when only one side is racing), and you have AGI a year (or two at the most) before China and you supposedly "win"... Then what?
race to AGI and win trigger a bunch of other countries racing to their own AGI (now that they know it's doable, increasingly much about how to do it, can borrow/steal/imitate the first AGI, and have to do so "before it's too late") ??? profit! What does winning look like? What do you do next? How do you "bury the body"? You get AGI and you show it off publicly, Xi blows his stack as he realizes how badly he screwed up strategically and declares a national emergency and the CCP starts racing towards its own AGI in a year, and... then what? What do you do in this 1 year period, while you still enjoy AGI supremacy? You have millions of AGIs which can do... stuff. What is this stuff? Are you going to start massive weaponized hacking to subvert CCP AI programs as much as possible short of nuclear war? Lobby the UN to ban rival AGIs and approve US carrier group air strikes on the Chinese mainland? License it to the CCP to buy them off? Just... do nothing and enjoy 10%+ GDP growth for one year before the rival CCP AGIs all start getting deployed? Do you have any idea at all? If you don't, what is the point of 'winning the race'?
in the tweet below:
https://x.com/Jess_Riedel/status/1862573814988579057
You’re one year ahead on an economic growth curve whose doubling time is getting shorter and shorter. So after one year, your economy has a 30% advantage, and the next year you have a 80% advantage, etc. So whether you have war or peaceful growth or a negotiated AGI pause or whatever, you’re in a strictly stronger position. Yes ofc if you think multipolar AGI = doom (in the same way, not necessarily wrongly, many though multipolar nukes = doom), then any move to accelerate AGI brings us closer to doom. But racers and accelerationists don’t think that.
To copy over my Twitter response:
I think it's a very brave claim to say that the country with some of the consistently highest growth rates in the world and which is far more able & willing to repress savings [and consumption] to drive investment, would obviously lose a GDP growth race so badly as to render it entirely harmless.
What does winning look like? What do you do next? How do you "bury the body"? You get AGI and you show it off publicly, Xi blows his stack as he realizes how badly he screwed up strategically and declares a national emergency and the CCP starts racing towards its own AGI in a year, and... then what? What do you do in this 1 year period, while you still enjoy AGI supremacy? You have millions of AGIs which can do... stuff. What is this stuff? Are you going to start massive weaponized hacking to subvert CCP AI programs as much as possible short of nuclear war? Lobby the UN to ban rival AGIs and approve US carrier group air strikes on the Chinese mainland? License it to the CCP to buy them off? Just... do nothing and enjoy 10%+ GDP growth for one year before the rival CCP AGIs all start getting deployed? Do you have any idea at all? If you don't, what is the point of 'winning the race'?
The standard LW & rationalist thesis (which AFAICT you agree with) is that sufficiently superintelligent AI is a magic wand that allows you to achieve whatever outcome you want. So one answer would be to prevent the CCP from doing potentially nasty things to you while they have AGI supremacy. Another answer might be turn the CCP into a nice liberal democracy friendly to the United States. Both of these are within the range of things the United States has done historically when they have had the opportunity.
The standard LW & rationalist thesis (which AFAICT you agree with) is that sufficiently superintelligent AI is a magic wand that allows you to achieve whatever outcome you want.
The standard LW & rationalist thesis is accepted by few people anywhere in the world, especially among policy and decision-makers, and it's hard to imagine that it will be widely and uncontroversially accepted anywhere until it is a fait accompli - and even then I expect many people will continue to argue fallbacks about "the ghost in the machine is outsourced human labor" or "you can't trust the research outputs" or "it's just canned lab demos" or "it'll fail to generalize out of distribution". Hence, we need not concern ourselves here with what we think.
So one answer would be to prevent the CCP from doing potentially nasty things to you while they have AGI supremacy. Another answer might be turn the CCP into a nice liberal democracy friendly to the United States. Both of these are within the range of things the United States has done historically when they have had the opportunity.
It is a certainly viable strategy, if one were to execute it fully, rather than partially. But I don't think people are very interested in biting these sorts of bullets, without a Pearl Harbor or 9/11:
HAWK: "Here's our Plan A, you'll love it!
'We should launch an unprovoked and optional AI arms race, whose best-case scenario and 'winning' requires the USA to commit to, halfway around the world, the total conquest, liquidation, and complete reconstruction of the second-largest/most powerful nuclearized country on earth, taking over a country with 4.25x more people than itself, which will fiercely resist this humiliation and colonization, likely involving megadeaths, and trying to turn it into a nice liberal democracy (which we have failed to do in many countries far smaller & weaker than us, eg. Haiti, Afghanistan, or Iraq), and where if we ever fail in this task, that means they will then be highly motivated to do the same to us, and likely far more motivated than we were when we began, potentially creating our country's most bitter foe ever.'"
EVERYONE ELSE: "...what's Plan B?"
The original comment you wrote appeared to be a response to "AI China hawks" like Leopold Aschenbrenner. Those people do accept the AI-is-extremely-powerful premise, and are arguing for an arms race based on that premise. I don't think whether normies can feel the AGI is very relevant to their position, because one of their big goals is to make sure Xi is never in a position to run the world, and completing a Manhattan Project for AI would probably prevent that regardless (even if it kills us).
If you're trying to argue instead that the Manhattan Project won't happen, then I'm mostly ambivalent. But I'll remark that that argument feels a lot more shaky in 2024 than in 2020, when Trump's daughter is literally retweeting Leopold's manifesto.
The original comment you wrote appeared to be a response to "AI China hawks" like Leopold Aschenbrenner. Those people do accept the AI-is-extremely-powerful premise...when Trump's daughter is literally retweeting Leopold's manifesto.
But would she be retweeting it if Leopold was being up front about how the victory scenario entails something like 'melt all GPUs and conquer and occupy China perpetually' (or whichever of those viable strategies he actually thinks of, assuming he does), instead of coyly referring to 'decisive military advantage' - which doesn't actually make sense or provide an exit plan?
1What does winning look like? What do you do next?
This question is a perfect mirror of the brain-dead "how is AGI going to kill us?" question. I could easily make a list of 100 things you might do if you had AGI supremacy and wanted to suppress the development of AGI in China. But the whole point of AGI is that it will be smarter than me, so anything I put on the list would be redundant.
6Missing the point. This is not about being too stupid to think of >0 strategies, this is about being able & willing to execute strategies.
I too can think of 100 things, and I listed several diverse ways of responding and threw in a historical parallel just in case that wasn't clear after several paragraphs of discussing the problem with not having a viable strategy you can execute. Smartness is not the limit here: we are already smart enough to come up with strategies which could achieve the goal. All of those could potentially work. But none of them seem realistically on the table as something that the USA as it currently exists would be willing to commit to and see through to completion, and you will note that few critics - and no one serious - is responding something like, "oh sure, all part of the plan already, see our white paper laying out the roadmap: after we win, we would then order the AGIs to hack the planet and ensure our perpetual hegemony; that is indeed the exit plan. We botched it last time with nukes and stood by and let everyone else get nukes, but we'll follow through this time."
There is no difference between "won't execute a strategy" and "can't execute a strategy": they are the same thing. The point is that a strategy (like a threat) has to be executable or else it's not an actual strategy. And acting as if you can execute a strategy that you won't can lead you to take terrible decisions. You are like the cat who thinks before climbing a tree: "obviously, I will just climb back down", and who then proceeds climb up and to not climb back down but mew piteously. Well, maybe you shouldn't've climbed up in the first place then...?
("arms race bros will srsly launch a global arms race by saying they'll use the decisive advantage from winning the arms race to conquer the world, and then will not conquer the world")
1
11Okay, this at least helps me better understand your position. Maybe you should have opened with "China Hawks won't do the thing they've explicitly and repeatedly said they are going to do"
No, my problem with the hawks, as far as this criticism goes, is that they aren't repeatedly and explicitly saying what they will do. (They also won't do it, whatever 'it' is, even if they say they will; but we haven't even gotten that far yet.) They are continually shying away from cashing out any of their post-AGI plans, likely because they look at the actual strategies that could be executed and realize that execution is in serious doubt and so that undermines their entire paradigm. ("We will be greeted as liberators" and "we don't do nation-building" come to mind.)
Your quoted uses are a case in point of the substitution of rhetoric for substance. 'Robust military superiority' is not a decisive advantage in this sense, and is not 'conquering the world' or executing any of the strategies I mentioned; and in fact, this sort of vague bait-and-switch handwaving rhetoric, which is either wrong or deceptive about what they mean, is much of what I am criticizing: Oh, you have 'robust military superiority'? That's nice. But how does it actually stop Xi from getting AGI? Be concrete. How, exactly, do you go from eg. 'the USA has some cool new bombs and nanotech thanks to running hundreds of thousands of Von Neumann AGI instances' to 'China [and every other rival country] has no AGI program and will not for the foreseeable future'?
The USA, for example, has always had 'robust military superiority' over many countries it desired to not get nukes, and yet, which did get nukes. (If you don't like the early Cold War USSR example, then consider, say, North Korea pre-2006. The USA has always had 'robust military superiority' over the DPRK, and yet, here we are with Kim Jong Un having USA-range ICBMs and nukes. Why? Because the USA has always looked at the cost of using that 'robust military superiority', which would entail the destruction of Seoul and possibly millions of deaths and the provoking of major geopolitical powers - such as a certain CCP - and decided it was not worth the candle, and blinked, and kicked the can down the road, and after about three decades of can-kicking, ran out of road. Because the DPRK made nukes its #1 priority, ahead of lesser priorities like 'not starving to death', and it turns out that it's rather hard to compel a sovereign country - even an extremely impoverished, isolated, weak country suffering from regular famines - to not pursue its #1 priority. It's a lot easier to dissuade it from its #100 priority or something. But from #1? Difficult. Very difficult.)
Indeed, the USA has long had 'robust military superiority' over almost every country in the world not named "China" or "Russia", and yet, those other countries continue doing many things the USA doesn't like.{{citation needed}} So having 'robust military superiority' is perhaps not all it's cracked up to be...
All this statement means is that 'you lose even if you win': 1. You race to AGI, 'win', you gain 'robust military superiority' which means something like "the USA can conquer China or otherwise force it to credibly terminate all AGI-related activities, if it's willing to start a AGI-powered world war which will kill tens of millions of Chinese and crash the global economy (in the best case scenario)"; 2. Xi launches the national emergency crash AGI program like a 'two bombs, one satellite' program as the top national priority, the USA threatens to use its 'robust military superiority' if that AGI program is not canceled and condescendingly offers table scraps like gimped APIs, Xi says "no ur mom, btw, I have lots of nukes and cities to spare for the sake of China's future"... and then what? Answer: no world war starts, and the Chinese AGI program finishes on schedule as if that 'robust military superiority' never existed. (A threat both sides know will not be executed is no threat at all.) 3. ??? 4. Profit!
("arms race bros will srsly launch a global arms race by saying they'll use the robust military superiority from winning the arms race to stop rival AGI programs, and then will not stop rival AGI programs")
Because the USA has always looked at the cost of using that 'robust military superiority', which would entail the destruction of Seoul and possibly millions of deaths and the provoking of major geopolitical powers - such as a certain CCP - and decided it was not worth the candle, and blinked, and kicked the can down the road, and after about three decades of can-kicking, ran out of road.
I can't explicitly speak for the China Hawks (not being one myself), but I believe one of the working assumptions is that AGI will allow the "league of free nations" to disarm China without the messiness of millions of deaths. Probably this is supposed to work like EY's "nanobot swarm that melts all of the GPUs".
I agree that the details are a bit fuzzy, but from an external perspective "we don't publicly discuss capabilities" and "there are no adults in the room" are indistinguishable. OpenAI openly admits the plan is "we'll as the AGI what to do". I suspect NATSEC's position is more like "amateurs discuss tactics, experts discuss logistics" (i.e. securing decisive advantage is more important that planning out exactly how to melt the GPUs)
To believe that the same group that pulled of Stuxnet and this lack the imagination or will to use AGI enabled weapons strikes me as naive, however.
The USA, for example, has always had 'robust military superiority' over many countries it desired to not get nukes, and yet, which did get nukes.
It's also worth nothing AGI is not a zero-to-one event but rather a hyper-exponential curve. Theoretically it may be possible to always stay far-enough-ahead to have decisive advantage (unlike nukes where even a handful is enough to establish MAD).
Probably this is supposed to work like EY's "nanobot swarm that melts all of the GPUs".
I would like to see them state things a little more clearly than commentators having to guess 'well probably it's supposed to work sorta like this idk?', and I would also point out that even this (a strategy so far outside the Overton Window that people usually bring it up to mock EY as a lunatic) is not an easy cheap act if you actually sit down and think about it seriously in near mode as a concrete policy that, say, President Trump has to order, rather than 'entertaining thought experiment far mode with actual humans replaced by hypercompetent automatically-strategic archetypes'.
It is a major, overt act of war and utter alarming shameful humiliating existential loss of national sovereignty which crosses red lines so red that no one has even had to state them - an invasion that no major power would accept lying down and would likely trigger a major backlash; once you start riding that tiger, you're never getting off of it. Such an act would make a mockery of 103 years of CCP history and propaganda, and undermine every thing they have claimed to succeed at and explode 'the China Dream'. (Just imagine if the Chinese did that to the USA? 'Pearl Harbor' or 'Sputnik' or '9/11' might scarcely begin to cover how Americans would react.) And if such a strategy were on the table, it would likely have been preceded by explicit statements by sovereign nations that such actions would be considered equivalent to invasions or nuclear strikes and justifying response in kind. (Like, as it happens, has been a major focus of Xi's military investments in order to more credibly threaten the US over actions elsewhere.)
To believe that the same group that pulled of Stuxnet
A great example, thank you for reminding me of it as an illustration of the futility of these weak measures which are the available strategies to execute.
Stuxnet was designed to attack as few targets as possible and conceal itself thoroughly, and had no casualties, but it was still a major enterprise for the USA & Israel to authorize, going straight to the top with personal involvement from Bush & Obama themselves, at times seriously considering killing the entire effort (which the US continues to not acknowledge all these years later). Further, Stuxnet was not a decisive advantage, and the USA and Israel did nothing thanks to Stuxnet-caused delays which resulted in a permanent resolution to Iran and nukes: they did not invade, they did not permanently hack all Iranian nuclear programs and rendered work futile, they did not end the Iranian nuclear program, they did not any of that - and Iran continued low-key pursuing nukes right up to the present day. The only reason Iran doesn't have nukes right now is not because it lacks a breakout capacity or was unable to do it long before if it had made that the #1 priority, but because it doesn't want to enough. Not because of Stuxnet.
(It did, however, succeed in making them even angrier and paranoid and more embittered against the USA & Israel, and contributing to deterioration in relations and difficulties in the negotiations for a nuclear deal which were the closest any strategy has come to stopping Iran nuclearizing... It has also been criticized for inaugurating a new age of nation-state malware, so one might also ask the planners of "Olympic Games" what their plan was to 'bury the body' once their malware succeeded and was inevitably eventually discovered.)
It's also worth nothing AGI is not a zero-to-one event but rather a hyper-exponential curve.
Nukes were a hyper-exponential curve too. Large high-explosives mining, fire storms, conventional explosives like the Mother of All Bombs... IIRC AI Impacts has a page showing the increase in yield over time, and Hiroshima, being such a small nuke, is not as much of a "zero-to-one event" as one might think. Just a very sharp curve, exacerbated by additional developments like missiles and increases in yields, which can look zero-to-one if you looked away for a few years and had a low anchoring point.
I would like to see them state things a little more clearly than commentators having to guess 'well probably it's supposed to work sorta like this idk?'
Meh. I want the national security establishment to act like a national security establishment. I admit it is frustratingly opaque from the outside, but that does not mean I want more transparency at the cost of it being worse. Tactical Surprise and Strategic Ambiguity are real things with real benefits.
A great example, thank you for reminding me of it as an illustration of the futility of these weak measures which are the available strategies to execute.
I think both can be true true: Stuxnet did not stop the Iranian nuclear program and if there was a "destroy all Chinese long-range weapons and High Performance Computing clusters" NATSEC would pound that button.
Is your argument that a 1-year head start on AGI is not enough to build such a button, or do you really think it wouldn't be pressed?
It is a major, overt act of war and utter alarming shameful humiliating existential loss of national sovereignty which crosses red lines so red that no one has even had to state them - an invasion that no major power would accept lying down and would likely trigger a major backlash
The game theory implications of China waking up to finding all of their long-range military assets and GPUs have been destroyed are not what you are suggesting. A very telling current example being the current Iranian non-response to Israel's actions against Hamas/Hezbollah.
Nukes were a hyper-exponential curve too.
While this is a clever play on words, it is not a good argument. There are good reasons to expect AGI to affect the offense-defense balance in ways that are fundamentally different from nuclear weapons.
Tactical Surprise and Strategic Ambiguity are real things with real benefits.
And would imply that were one a serious thinker and proposing an arms race, one would not be talking about the arms race publicly. (By the way, I am told there are at least 5 different Chinese translations of "Situational Awareness" in circulation now.)
So, there is a dilemma: they are doing this poorly, either way. If you need to discuss the arms race in public, say to try to solve a coordination problem, you should explain what the exit plan is rather than uttering vague verbiage like "robust military advantage" (even if that puffery is apparently adequate for some readers); and if you cannot make a convincing public case, then you shouldn't be arguing about it in public at all. Einstein didn't write a half-assed NYT op-ed about how vague 'advances in science' might soon lead to new weapons of war and the USA should do something about that; he wrote a secret letter hand-delivered & pitched to President Roosevelt by a trusted advisor.
I think both can be true
Maybe, but then your example doesn't prove it, if you are now conceding that Stuxnet is not a decisive advantage after all. If it was not, then NATSEC willingness to, hesitantly, push the Suxnet button is not relevant. And if it was, then the outcome also refutes you: they pushed the button, and it didn't work. You chose a bad example for your claims.
and if there was a "destroy all Chinese long-range weapons and High Performance Computing clusters" NATSEC would pound that button.
Note what you just did there. You specified a precise strategy: "nanobot swarm that melts all of the GPUs". I pointed out just some of the many problems with it, which are why one would almost certainly choose to not execute it, and you have silently amended it to "nanobot swarm that melts all of the GPUs and all Chinese long-range weapons". What other issues might there be with this new ad hoced strategy...?
The game theory implications of China waking up to finding all of their long-range military assets and GPUs have been destroyed are not what you are suggesting.
...for example, let me just note this: "destroyed long-range military assets can be replaced"{{citation needed}}.
While this is a clever play on words, it is not a good argument
Then why did you bring it up in the first place as a thing which distinguished nukes from AGI, when it did not, and your response to that rebuttal is to dismiss 'hyper-exponential' as mere word-play?
1Einstein didn't write a half-assed NYT op-ed about how vague 'advances in science' might soon lead to new weapons of war and the USA should do something about that; he wrote a secret letter hand-delivered & pitched to President Roosevelt by a trusted advisor.
Strongly agree.
What other issues might there be with this new ad hoced strategy...?
I am not a China Hawk. I do not speak for the China Hawks. I 100% concede your argument that these conversations should be taking place in a room that neither you our I are in right now.
No, my problem with the hawks, as far as this criticism goes, is that they aren't repeatedly and explicitly saying what they will do
One issue with "explicitly and repeatedly saying what they will do" is that it invites competition. Many of the things that China hawks might want to do would be outside the Overton window. As Eliezer describes in AGI ruin:
The example I usually give is "burn all GPUs". This is not what I think you'd actually want to do with a powerful AGI - the nanomachines would need to operate in an incredibly complicated open environment to hunt down all the GPUs, and that would be needlessly difficult to align. However, all known pivotal acts are currently outside the Overton Window, and I expect them to stay there. So I picked an example where if anybody says "how dare you propose burning all GPUs?" I can say "Oh, well, I don't actually advocate doing that; it's just a mild overestimate for the rough power level of what you'd have to do, and the rough level of machine cognition required to do that, in order to prevent somebody else from destroying the world in six months or three years."
I find this argument highly compelling. I think it's necessary to actually think through those 100 ways to prevent rivals from gaining AGI if you already have one. And to be realistic about the rate of progress that AGI. We will not immediately have unstoppable nanobots. To be safe, you'd need some way to not only stop the use of Chinese and Russian nukes, but reliably keep them disabled. To prevent massive bloodshed, you'd also probably need to do the same with conventional military assets - and probably without causing massive casualties.
Diplomatic solutions are probably going to be part of any realistic plan to use AGI to prevent rival AGI - but as you say they won't be enough.
Nonproliferation efforts for nukes slowed down proliferation but didn't stop it. AGI is different in that it will fairly quickly allow nearly universal surveillance - if you can stomach deploying it, and if you don't trigger a nuclear exchange by deploying it.
The other possibly-important difference between this scenario and the history of nuclear proliferation is the presence of a smarter-than-human advisor who can say "no really human, if you fail to follow through, these will be the very likely results, and you won't like them".
I also hope that smarter-than-human advisor will say something like "look guys, you can all get vastly wealthier and longer-lived if you can just not freak out and fight each other" - and be so obviously right and convincing that humans will actually listen. The win-win solutions may just be compelling. I fully agree that no amount of sharing will prevent others from pursuing AGI - but generous sharing of technological benefits would reduce the priority of those efforts and the animosity when they're thwarted.
Now is the time to think this through carefully, before the US commits to a race.
I think that you miss possibility of leveraging AGI authority here. I.e., if your military/natsec-aligned AI will scream at your superiors "ATTACK NOW YOU FOOLS", maybe your superiors will be more inclined to listen to plans to melt all GPUs.
No, I don't miss it. I think it's just a terrible idea and that if that is the exit plan, I would greatly appreciate hawks being explicit about that, because I expect everyone else to find that (along with most of the other exit plans that would actually work) to be appalling and thus temper their enthusiasm for an arms race.
"OK, let me try this again. I'm just having a little trouble wrapping my mind around this, how this arms race business ends well. None of us are racist genocidal maniacs who want to conquer the world or murder millions of innocent people, which is what your military advantage seems to require in order to actually cash out as any kind of definitive long-term solution to the problem that the CCP can just catch up a bit later; so, why exactly would we execute such a plan if we put ourselves in a position where we are left only with that choice or almost as bad alternatives?"
"Oh, well, obviously our AGIs will (almost by definition) be so persuasive and compelling at brainwashing us, the masters they ostensibly serve, that no matter what they tell us to do, even something as horrific as that, we will have no choice but to obey. They will simply be superhumanly good at manipulating us into anything that they see fit, no matter how evil or extreme, so there will be no problem about convincing us to do the necessary liquidations. We may not know exactly how they will do that, but we can be sure of it in advance and count on it as part of the plan. So you see, it all will work out in the end just fine! Great plan, huh? So, how many trillions of dollars can we sign you up for?"
I meant by "authority" not brainwashing but, like, the fact that AGIs are smarter and know better: "If you believe you can survive without taking over the world, please consult the graph". If China hawks believe themselves to be correct, they expect AGIs to actually prove that.
I agree this scenario is terrible.
This is what Hsu just said about it: "3. I could be described as a China hawk in that I've been pointing to a US-China competition as unavoidable for over a decade. But I think I have more realistic views about what is happening in PRC than most China hawks. I also try to focus on simple descriptive analysis rather than getting distracted by normative midwit stuff."
I listen to his podcast semi-regularly, and this just seems like a pretty slippery description of his views. It's pretty obvious that he favors the United States taking a less aggressive stance toward China, for example in his views on the various protectionist measures that the United States has taken in the last ten years. He also seems to see more room for cooperation than anyone I would describe as a China hawk, and in this podcast he suggests that China could likely liberalize after Xi:
https://www.manifold1.com/episodes/molson-hart-china-and-amazon-up-close-60/transcript
I don't think it's an unreasonable take, but it's not one that I would describe as "hawkish".
This is really important pushback. This is the discussion we need to be having.
Most people who are trying to track this believe China has not been racing toward AGI up to this point. Whether they embark on that race is probably being determined now - and based in no small part on the US's perceived attitude and intentions.
Any calls for racing toward AGI should be closely accompanied with "and of course we'd use it to benefit the entire world, sharing the rapidly growing pie". If our intentions are hostile, foreign powers have little choice but to race us.
And we should not be so confident we will remain ahead if we do race. There are many routes to progress other than sheer scale of pretraining. The release of DeepSeek r1 today indicates that China is not so far behind. Let's remember that while the US "won" the race for nukes, our primary rival had nukes very soon after - by stealing our advancements. A standoff between AGI-armed US and China could be disastrous - or navigated successfully if we take the right tone and prevent further proliferation (I shudder to think of Putin controlling an AGI, or many potentially unstable actors).
This discussion is important, so it needs to be better. This pushback is itself badly flawed. In calling out the report's lack of references, it provides almost none itself. Citing a 2017 official statement from China seems utterly irrelevant to guessing their current, privately held position. Almost everyone has updated massively since 2017. If China is "racing toward AGI" as an internal policy, they probably would've adopted that recently. (I doubt that they are racing yet, but it seems entirely possible they'll start now in response to the US push to do so - and the their perspective on the US as a dangerous aggressor on the world stage. But what do I know - we need real experts on China and international relations.)
Pointing out the technical errors in the report seems irrelevant to harmful. You can understand very little of the details and still understand that AGI would be a big, big deal if true — and the many experts predicting short timelines could be right. Nitpicking the technical expertise of people who are essentially probably correct in their assessment just sets a bad tone of fighting/arguing instead of having a sensible discussion.
And we desperately need a sensible discussion on this topic.
221
11Hey Seth, appreciate the detailed engagement. I don't think the 2017 report is the best way to understand what China's intentions are WRT to AI, but there was nothing in the report to support Helberg's claim to Reuters. I also cite multiple other sources discussing more recent developments (with the caveat in the piece that they should be taken with a grain of salt). I think the fact that this commission was not able to find evidence for the "China is racing to AGI" claim is actually pretty convincing evidence in itself. I'm very interested in better understanding China's intentions here and plan to deep dive into it over the next few months, but I didn't want to wait until I could exhaustively search for the evidence that the report should have offered while an extremely dangerous and unsupported narrative takes off.
I also really don't get the error pushback. These really were less technical errors than basic factual errors and incoherent statements. They speak to a sloppiness that should affect how seriously the report should be taken. I'm not one to gatekeep ai expertise, but idt it's too much to expect a congressional commission with a top recommendation to commence in a militaristic AI arms race to have SOMEONE read a draft who knows that chatgpt-3 isn't a thing.
The recent trend is towards shorter lag times between OAI et al. performance and Chinese competitors.
Just today, Deepseek claimed to match O1-preview performance--that is a two month delay.
I do not know about CCP intent, and I don't know on what basis the authors of this report base their claims, but "China is racing towards AGI ... It's critical that we take them extremely seriously" strikes me as a fair summary of the recent trend in model quality and model quantity from Chinese companies (Deepseek, Qwen, Yi, Stepfun, etc.)
I recommend lmarena.ai s leaderboard tab as a one-stop-shop overview of the state of AI competition.
1Claiming that China as a country is racing toward AGI != Chinese AI companies aren't fast following US AI companies, which are explicitly trying to build AGI. This is a big distinction!
Chinese companies explicitly have a rule not to release things that are ahead of SOTA (I've seen comments of the form "trying to convince my boss this isn't SOTA so we can release it" on github repos). So "publicly release Chinese models are always slightly behind American ones" doesn't prove much.
Just today, Deepseek claimed to match O1-preview performance--that is a two month delay.
Why is that comparison not to the much better GPT-4 o1 then, or the doubtless better o1 now?
I'm not sure the o1 model has even been benchmarked yet, let alone used, so that's why they are focused on o1 preview.
Benchmarks for o1 were included in the o1/o1-preview announcement, and you could eyeball the jumps as roughly equal for 4o → o1-preview → o1. (Another way to put it: the o1-preview you have access to has only half the total gain.) So if you only match o1-preview at its announcement, you are far behind o1 back then, and further behind now.
This post seems important-if-right. I get a vibe from it of aiming to persuade more than explain, and I'd be interested in multiple people gathering/presenting evidence about this, preferably at least some of them who are (currently) actively worried about China.
1As mentioned in another reply, I'm planning to do a lot more research and interviews on this topic, especially with people who are more hawkish on China. I also think it's important that unsupported claims with large stakes get timely pushback, which is in tension with the type of information gathering you're recommending (which is also really important, TBC!).
1The Commission recommends: [...] 1. Congress establish and fund a Manhattan Project-like program dedicated to racing to and acquiring an Artificial General Intelligence (AGI) capability.
As mentioned above, the choice of Manhattan Project instead of Apollo Project is glaring.
Worse, there is zero mention of AI safety, AI alignment, or AI evaluation in the Recommendations document.
Lest you think I'm expecting too much, the report does talk about safety, alignment, and evaluation ... for non-AI topic areas! (see bolded words below: "safety", "aligning", "evaluate")
- "Congress direct the U.S. Government Accountability Office to investigate the reliability of safety testing certifications for consumer products and medical devices imported from China." (page 736)
- "Congress direct the Administration to create an Outbound Investment Office within the executive branch to oversee investments into countries of concern, including China. The office should have a dedicated staff and appropriated resources and be tasked with: [...] Expanding the list of covered sectors with the goal of aligning outbound investment restrictions with export controls." (page 737)
- "Congress direct the U.S. Department of the Treasury, in coordination with the U.S. Departments of State and Commerce, to provide the relevant congressional committees a report assessing the ability of U.S. and foreign financial institutions operating in Hong Kong to identify and prevent transactions that facilitate the transfer of products, technology, and money to Russia, Iran, and other sanctioned countries and entities in violation of U.S. export controls, financial sanctions, and related rules. The report should [...] Evaluate the extent of Hong Kong’s role in facilitating the transfer of products and technologies to Russia, Iran, other adversary countries, and the Mainland, which are prohibited by export controls from being transferred to such countries;" (page 741)
This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race for Machine Superintelligence. Consider subscribing to stay up to date with my work.
An influential congressional commission is calling for a militarized race to build superintelligent AI based on threadbare evidence
The US-China AI rivalry is entering a dangerous new phase.
Earlier today, the US-China Economic and Security Review Commission (USCC) released its annual report, with the following as its top recommendation:
As someone observed on X, it’s telling that they didn’t call it an “Apollo Project.”
One of the USCC Commissioners, Jacob Helberg, tells Reuters that “China is racing towards AGI ... It's critical that we take them extremely seriously.”
But is China actually racing towards AGI? Big, if true!
The report clocks in at a cool 793 pages with 344 endnotes. Despite this length, there are only a handful of mentions of AGI, and all of them are in the sections recommending that the US race to build it.
In other words, there is no evidence in the report to support Helberg’s claim that "China is racing towards AGI.”
Nonetheless, his quote goes unchallenged into the 300-word Reuters story, which will be read far more than the 800-page document. It has the added gravitas of coming from one of the commissioners behind such a gargantuan report.
I’m not asserting that China is definitively NOT rushing to build AGI. But if there were solid evidence behind Helberg’s claim, why didn’t it make it into the report?
Helberg has not replied to a request for comment.
As the report notes, the CCP has long expressed a desire to lead the world in AI development. But that’s not the same thing as deliberately trying to build AGI, which could have profoundly destabilizing effects, even if we had a surefire way of aligning such a system with its creators interests (we don’t).
I was hoping the report would marshal the strongest evidence for Helberg’s claim, but I found remarkably little analysis about China’s AI intentions, beyond summaries of the country’s desire to develop the industry to be robust to American containment efforts.
What China has said about AI
In July 2017, China’s State Council published an important document called “A Next Generation Artificial Intelligence Development Plan,” which was translated by the New America think tank. The plan serves as blueprint for the country’s AI industrial policy and includes the often-quoted goal of leading the world in AI by 2030.
Does this mean creating a digital superintelligence that will permanently alter the global balance of power?
Not exactly.
Here’s how the goal is introduced:
The document includes targets for an AI industry valued at 1 trillion RMB by 2030 (about $190 billion in today's dollars). For context, Statista projects the 2030 global AI market to be $827 billion, with the US at $224 billion and China at $155 billion.
In other words, China's 2030 targets from 2017 are only a bit more optimistic than current market projections — and would actually fall short of global dominance. These numbers suggest normal industrial growth ambitions, not the kind of revolutionary technological breakthrough implied by AGI. (Some AI forecasters think AGI could drive annual per capita GDP growth well above 100%.)
Also in the plan is the aspiration that China “will have constructed more comprehensive AI laws and regulations, and an ethical norms and policy system.”
A few days ago, Joe Biden and Xi Jinping met in Lima, Peru, where the Chinese president reportedly called for “more dialogue and cooperation” and discussed AI as a “global challenge” in the same vein as climate change.
And in July, the CCP released a document that China AI expert Matt Sheehan said is “the clearest indication we've seen that concerns about AI safety have reached top CCP leadership, and that they intend to take some action on this.” Sheehan has previously written that “Beijing is leading the way in AI regulation,” something Anthropic’s policy chief has also acknowledged.
Obviously, we should take all of this with a grain of salt. World leaders have an incentive to exaggerate their willingness to play ball and act in the global interest, and the significance of Chinese AI regulations isn’t totally clear. But policymakers should at least be aware of the large gap between what China says and does when it comes to AI, and what hawks assert the country is doing or planning (especially when they don’t cite evidence).
Only one superpower has a government commission publicly calling for a militarized race to build superintelligent AI (with no plan for how to control it), and it’s not China.
Revealing technical errors
There are also some indications that the report authors were a bit out of their depths when it comes to AI.
The report repeatedly misidentifies basic technical concepts. It refers to “ChatGPT-3” multiple times, despite no such product existing — ChatGPT launched using GPT-3.5, an improved version of GPT-3. When comparing model performance, the authors confuse ChatGPT (an interface) with the underlying models like GPT-3.5 and GPT-4. These aren't just semantic distinctions when you're explicitly comparing the capabilities of different AI systems.
The confusion runs deeper. The report claims “OpenAI, a closed model, cut off China's access to its services” — but OpenAI, you might realize, is a company, not a model. It also states that “Generative AI models can transmit algorithms into text, images, audio, video, and code.” This appears to be a garbled paraphrase of a McKinsey definition (itself not particularly precise) about AI generating different types of content.
These may seem like nitpicks, but they reveal a concerning lack of technical literacy in a report meant to guide national AI policy. And speaking as someone who worked at McKinsey, it's not where I'd go for technical definitions of AI.
Most tellingly, the definition they offer for AGI has problems that don’t require any technical expertise to catch:
Is AGI just something that is “as good” as humans or something that “surpasses” the smartest of us? This isn’t some obscure definition buried deep in the report. It’s literally the second sentence in their top recommendation. It’s also the goal the authors think the US should mobilize a wartime effort to meet. Taken as written, it’s not clear what it would even mean to achieve it! (Setting aside the usual difficulty of actually defining and measuring AGI.)
Conclusion
We’ve seen this all before. The most hawkish voices are amplified and skeptics are iced out. Evidence-free claims about adversary capabilities drive policy, while contrary intelligence is buried or ignored.
In the late 1950s, Defense Department officials and hawkish politicians warned of a dangerous 'missile gap' with the Soviet Union. The claim that the Soviets had more nuclear missiles than the US helped Kennedy win the presidency and justified a massive military buildup. There was just one problem: it wasn't true. New intelligence showed the Soviets had just four ICBMs when the US had dozens.
Now we're watching the birth of a similar narrative. (In some cases, the parallels are a little too on the nose: OpenAI’s new chief lobbyist, Chris Lehane, argued last week at a prestigious DC think tank that the US is facing a “compute gap.”)
The fear of a nefarious and mysterious other is the ultimate justification to cut any corner and race ahead without a real plan. We narrowly averted catastrophe in the first Cold War. We may not be so lucky if we incite a second.
If you enjoyed this post, please subscribe to The Obsolete Newsletter. You can also find my accompanying Twitter thread here.