Why is it worse for x risk for China to win the AI race?

My understanding of the standard threat model is that, at some point, governments will need to step in and shut down or take control over profitable and popular projects for the good of all society. I look at China, and I look at the US, and I can't say "the US is the country I would bet on to hit the big red button here".

There's got to be something I'm missing here.

My point is more that we have millennia of experience building tools and social structures for making humans able to successfully accomplish tasks, and maybe 2 years of experience building tools and structures for making LLM agents able to successfully accomplish tasks.

I do agree that there's some difference in generality, but I expect that if we had spent millennia gathering experience building tools and structures tailored towards making LLMs more effective, the generality failures of LLMs would look a lot less crippling.

If you take a bunch of LLMs and try to get them to collaboratively build a 1GW power plant, they are going to fail mostly in ways like

1. they have hilariously poor vision
2. they don't make effective use of new tools
3. they don't create new tools to trivialize repetitive tasks
4. they get caught in loops of trying the same ineffective thing over and over

All of these are failure modes which can be substantially mitigated by better scaffolding of the sort that is hard to design in one shot but easy to iteratively improve over time.

Humans are hilariously bad at wilderness survival in the absence of societal knowledge and support. The support doesn't need to be 21st-century-shaped but we do need both physical and social technology to survive and reproduce reliably.

That doesn't matter much, though, because humans live in an environment which contains human civilization. The "holes" in our capabilities don't come up very often.

The right tools could also paper over many of the deficiencies of LLM agents. I don't expect the tools which make groups of LLM agents able to collectively do impressive things to result in particularly human-shaped agents though.

Concretely, sample efficiency is very important if you want a human-like agent that can learn on the job in a reasonable amount of time. It's much less important if you can train once on how to complete each task with a standardized set of tools, and then copy the trained narrow system around as needed.

(Note: perhaps I should say "language-capable agent" rather than "llm-based agent")

I think this is a decently engaging story, but it sounds like a Claude story, not a Tomás B story. Ending is too happy, technology is allowed to be good, and there are no themes of the protagonist being complicit in a system they find abhorrent. Also "the protagonist of these stories in my context window goes to therapy and resolves their internal tensions" is the most Claude it is possible for a story to be.

I would be sad if you stopped writing stories because other humans could write stories that are of similar quality by some metrics, and I will also be sad if you stop writing because AI can write fiction which is good in different ways to the ways your fiction is good.

I no longer consider scaffolded LLMs as a relevant concern/threat.

I am extremely surprised to see you say that, to the point that I think I must be misinterpreting you. What tools an LLM has the ability to use seems to have huge effects on its ability to do things.

Concretely, Claude 3.5 Sonnet can do far more useful coding tasks with a single tool to execute bash commands on a VM than Claude 4.5 Sonnet can in the absence of that tool. Or is "while loop plus tools" not the type of scaffolding you're referring to?

I think it's likely that without a long (e.g. multi-decade) AI pause, one or more of these "non-takeover AI risks" can't be solved or reduced to an acceptable level.

I think it is also worth considering the possibility that these risks aren't the sort of thing which can be reduced to an acceptable level with a decade-scale AI pause either. Particularly the ones which people have been trying to solve for centuries already (e.g. principal-agent problem).

In my experience using the LLM wrapper IDEs (cursor, windsurf, etc), if I ask the model to do some task where one of the assumptions I was making when writing the task was wrong (e.g. I ask it to surface some piece of information to the user in the response to some endpoint, but that piece of information doesn't actually exist until a later step of the process), GPT-5 will spin for a long time and go off and do stuff to my codebase until it gets some result which looks like success if you squint, while Sonnet 4.5 will generally break out of the loop and ask me for clarification.

Sonnet 4.5's behavior is what I want as a user but probably scores worse on the METR benchmark.

My best guess is it takes 5ish years to replicate everything in a machine shop minus the things which can't realistically be made in a machine shop (e.g. electronics, high speed steel stuff, diamond powder, maybe bearings). Much of that time would be spent on repetitive tasks like making screws. Mining and forestry robots would slow down the process more, likely quite a bit more, not so much because they're difficult as because they have a lot of parts.

AI development feels more similar to biology than to chemistry. Bright 11th graders shouldn't be doing experiments on culturing some previously unculturabke pathogen which would be a good bioweapon target and discussing their results, since the field is wide and shallow and it's not entirely impossible that their experiments are novel. On the other hand, if they're running basic experiments on culturing some specific common bacterium (e.g. e coli) better, they probably don't need to worry about accelerating bioweapon development even if there is a chance of them making a slight advancement to the field of biology as a whole.

The nanogpt speedrun feels more like developing better methods to culture e coli at a hobbyist level, and quite unlikely to lead to any substantial advancement applicable to the operational efficiency of well-funded companies at the frontier. Still, it probably is worth keeping track of when the work you're doing approaches the "this is actually something novel the frontier labs might use" mark, particularly if it's something more substantial than "here's how to use the hardware more efficiently to train this particular model".

In retrospect, sure, MAD worked out for us. But in 1899, Ivan Bloch asserted

... if any attempt were made to demonstrate the inaccuracy of my assertions by putting the matter to a test on a great scale, we should find the inevitable result in a catastrophe which would destroy all existing political organization. Thus, the great war cannot be made, and any attempt to make it would result in suicide.

This was before both world wars. After the first world war but before the second, others made similar arguments. In von Neumann's time, that argument did not have a good empirical track record, and his work on game theory gave him theoretical reasons not to expect the prediction of peace through MAD to hold. If there was something he was missing in 1948, it is not obvious what.