This is a special post for quick takes by [anonymous]. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Possible counterarguments:
It doesn't increase the risk as agents with nuclear arsenals already exist?
Current US government exploited unique resources - land etc
Current US government will oppose a new similar to US government organization to appear
P1 should be that existential risk is already high. For example moving from 70 per cent to 80 per cent is not large increase. Nuclear risk used to be very serious during previous cold war.
The main problem of your argument is that it is not clear why intelligent agent should use nukes against humanity, as it will terminate the agent too. To address this I suggest adding "creating independent robotic infrastructure" as a necessary condition. I have a post about it.
Also, from you argument follows that merging of AI and US government is the most natural path to AI dominance. I think it is true. But it is not generally accepted.
The way you word the second type might be working against you. ‘Updating’ brings to mind the computer function which neatly and quickly fills a bar and then the needed changes are complete. The human brain doesn’t work like that. To build new intuitions you need spaced repetition and thoughtful engagement with the belief you want to internalize. Thinking does work, you just can’t force it.
I’ll try my best, I’m by no means an expert. I don’t think there’s a one size fits all answer but let’s take your example of relationship between IQ and national prosperity. You can spend time researching what makes up prosperity and where that intersects with IQ and find different correlates between IQ and other attributes in individuals (the assumption being that individuals are a kind of unit to measure prosperity).
You can use spaced repetition to avoid burnout and gain fresh perspectives. The point is to build mental muscle memory and intuition on what moves the needle of prosperity. You might find, for example, that different contexts affect the relationship differently. So what you’re doing is not updating a belief or fact but rather improving the mental tool set used to analyze the world around us and arrive at beliefs.
It’s difficult because reality is often counterintuitive but we can improve our intuition. I’m sure there are better ways to describe the process. The idea that “the brain thinks new beliefs into itself” also feels crude and incomplete.