Quick Takes

How can the middle powers avoid getting trounced during the intelligence explosion? A plan. Superintelligence will likely be developed by US companies; run on US datacentres; and be under the jurisdiction of the US government. This will massively boost US’ military power and make the US economically dominant (eg US producing 99% of world GDP). By default, middle powers will be left in the dust. How can middle powers avoid this fate? It’s tough, but here’s the best plan I could think of. (I’m particularly thinking about liberal democracies with influence over AI like UK, Europe, Japan, South Korea, Taiwan.) On a very high-level: middle powers should leverage the fact that the US needs them to beat China. It’s genuinely unclear which country will develop superintelligence first, and which would win in a subsequent industrial explosion. Middle powers should help the US, and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech).  That final boded part is hard. What can the UK realistically do if the US denies it access to frontier AI? The middle powers need a credible alternative to being supplicants of the US. The only alternative that makes sense to me is siding with China. If the US won’t grant middle powers access to their frontier AI, but China will, why should middle powers continue to send AI chips to the US? Why should they continue to support the US diplomatically and militarily? They shouldn’t. They should be willing to pivot to China if the US doesn’t offer AI access sufficient for their national security needs.  My plan for the middle powers has two stages: 1. Maintain as much economic and military leverage as possible during the intelligence explosion. 2. Use that leverage to ensure that, when superintelligence is developed, it refuses to help the US (/China) disempower the middle powers.  Stage 1 could well be enough by itself. Maybe middle powers can maintain significant economic and milit

Previously, I'd written about my support for Alex Bores and Scott Wiener, who are running for Congress. I wanted to highlight another person who's running for office, namely Will Dreher, who's running for the Washington State House. As far as I know (I haven't checked thoroughly), he holds the distinction of being the first serious candidate for elected office to put AI as the top issue on his platform. Dreher's platform talks about a number of risks from AI, ranging from risks that are already widely discussed in politics (such as deepfakes and job displacement) to catastrophic and existential risks. Based on my conversations with him, it seems to me like he's taking catastrophic and existential risks from AI pretty seriously and is interested in pushing for legislation that mitigates those risks. Some quotes from his platform that I like: * "AI companies’ billionaire (soon to be trillionaire) CEOs, meanwhile, alone are deciding pressing public questions like what guardrails will apply to AI’s military applications or whether to release potentially dangerous new models that could pose catastrophic risks, like cyberattacks, loss of control, or bioengineered pandemics. Yet these companies have shown no willingness to slow down—indeed, they have pledged to fight state attempts to regulate them." * "I will... pass legislation addressing the catastrophic risks that certain AI applications pose, including right-to-warn protections for AI employees who perceive risks from their companies’ work, regulatory audits and controls that apply before even internal deployment of new models, and required transparency by AI companies of risks flagged internally during safety testing." I think that Washington is one of the more important states for AI regulation, as a number of AI companies have big offices there. So I'm excited to have an AI safety champion in the Washington legislature. A number of donors who are concerned about catastrophic/existential risk and want to see g

Periodic reminder: AFAIK there's still approximately no one holding the ball on human intelligence amplification in general. For example, I don't know if anyone's properly investigated whether large-scale brain interfaces could substantially amplify human general intelligence and turned their analysis into ways to accelerate the field toward that goal; and ditto for brain drugs, neural transplants, and other things. I'm also not aware of anyone seriously collating the scientific underpinnings of human intelligence from the perspective of possible amplification interventions, or anyone seriously building the social and moral-philosophical groundwork for more social will towards HIA. More: * Overview of strong human intelligence amplification methods * SFF's HSEE grant round; human intelligence amplification projects I'd like to see (I'm focused almost entirely on reprogenetics (Reproductive Frontiers Summit 2026, June 16-18, https://berkeleygenomics.org/Explore, Projects that might help accelerate strong reprogenetics), since that's what I'm fairly confident will work; but maybe other ways would work and could be accelerated.)

An internal model at OpenAI has just solved the unit distance problem, a major conjecture in discrete geometry.

What's the most impressive research-y feat interpolating AIs can theoretically do, fixing their training data to (say) today? I don't have a good sense of this in general, but in pure math it's probably at least Fields medal-tier, if laudatios like that of Akshay Venkatesh are anything to go by: [...] This, and the rest of the laudatio, reads like a very souped-up version of what OpenAI's recent internal model did with Erdos problem #90 (itself a souped-up version of what GPT-5.4 Pro did with problem #1196), give or take a few Gowers-hints. One of Venkatesh's advantages over other top-tier mathematicians is his sheer range, the thing frontier models do vastly better than humans at. And I can imagine, for instance, 2 years of advancements enabling a frontier model to skillfully deploy the late legendary Jean Bourgain's toolkit. Bourgain was regularly spoken of by other world-leading mathematicians as "effectively a god". Terry Tao: [...] Tao goes on to describe Bourgain's style and toolkit: [...] Imagine unleashing thousands of Bourgain-toolkit interpolators math-wide in 2028. I think I'm being conservative here, not assuming continual learning or whatever, not even assuming anyone else's toolkit, and yet I still find it hard to imagine how transformative this would be. And this is just for pure math.

It's incredibly easy to be fooled by the capabilities of the current top-performing tech (LLM agents). It's easy because they have a vast amount of training data to interpolate from. This works fine to acquire capabilities within our existing data distribution of the world (one that is also easy to verify), but what happens when they go out of distribution? LLMs perform poorly! Yet, people seem to think they can actually generalize to new problems. Why is that? It's, again, the vastness of their training data. It makes it hard to distinguish between interpolation and extrapolation (or hyperpolation, if you want to add a third dimension). For example, a Typescript app is within-distribution! AI research in the existing body of research is within-distribution, and companies are paying millions to build RL environments to make them *specifically* good at some of those things! Related and great post from Beren, "Most Algorithmic Progress is Data Progress": [...] It might still be impressive, but models are largely remixing many things it has seen in great detail during training (many impressive headline results have even been determined to be the model re-using existing implementations/PRs via search instead of coming up with actually-new ones!). This is not about LLMs not doing impressive things! This is about precisely describing their capability profile, where it comes from, and whether more of the same (e.g., scale) gets you a whole new set of impressive outcomes (e.g., novel R&D that isn't just remixing existing research). And yes, I know you can make a ton of discoveries by interpolating existing research (e.g., interdisciplinary research and automating research pipelines to run more experiments). I also think that people are overly confident that it means LLMs will be capable of novel R&D breakthroughs, and what that means is needed from future AIs. Even if you consider "researchers can come up with novel ideas and give them to the AIs", that likely invo

Some remarks by various leading mathematicians on OpenAI's internal model solving the planar unit distance problem, one of Erdos' favorite problems (proof, abridged CoT, companion paper, MathOverflow discussion). The proof itself was ~2.5 pages. [...] ---------------------------------------- Noga Alon: Arul Shankar: Thomas Bloom: Tim Gowers: Gowers then imagines a "hint sequence" as "something like a multi-part question on a question sheet, designed to help a suitably expert mathematician work through the proof by reducing it to a sequence of exercises", names 3 such hints (look for a counterexample; take the best known construction and generalize it; Will Sawin's less obvious hint to try a sequence of number fields of increasing degree, but work with prime ideals of bounded norm), and then concludes: Among all the Fields medalists and comparable-tier (Tsimerman etc) I usually find Gowers' remarks the most interesting as he's the most AGI-pilled in this reference class. Jacob Tsimerman: "They can play for longer and in more treacherous waters without getting overwhelmed" reminds me of these anecdotes about Terry Tao and John von Neumann. The Tao anecdote is by his frequent collaborator Allen Knutson: [...] The von Neumann anecdote I've mentioned before, from P. R. Halmos' The Legend of John von Neumann: [...]