AI is capable of revolutionary advances in mathematics. Machine learning research is not different in kind.
No, it's different in kind:
- ML research is not nearly as verifiable (in Lean or otherwise) as mathematics is, substantially reducing the effectiveness of RL on it[1]
- ML research takes much more time to perform and evaluate, and often costs even more than the Navier-Stokes swarm did
To take your own example, Anthropic's latest research shows that Claude is leading (not autonomously running) 26% of their AI model R&D work as of August 2026. Though that’s increasing pretty quickly, it shouldn't hit 100% before Christmas. In contrast, I think we can safely say that Claude is handling much more than that percentage of their mathematics work (such as it is). We can also see that while mathematics capabilities have advanced tremendously recently, other capabilities such as writing have not advanced as much. There is not full generalization out-of-distribution here. Capabilities remain spiky.
Now, that said, in principle, yes ASI is possible by Christmas. Slow takeoff is not a promise, even if it's been true so far. I think you're right that continual learning, or some comparable research breakthrough, would be the most likely candidate, and it could happen at any time.
I think you're also right that organized swarms of AI agents could contribute to faster-than-usual capabilities gains, but having looked into that a bunch in the past few days, it's not clear to me that we are yet in a place where coordinating swarms meaningfully increase effective compute over parallel agent work. Noam Brown at OpenAI said on Dwarkesh's podcast today that they aren't. Rather, they're still just a way to spend more money to get results faster. That does have RSI implications, but under current limited compute budgets.
- ^
Edit: Though see counterargument here that mathematics capabilities gains are mostly improved pretraining on good source material. Nevertheless, as that post points out, machine learning research does not have the same volume of quality material to work from as math does (imo, partly because math is so much more verifiable).
I think a good example is the Noam Brown interview with Dwarkesh; they're not compute constrained on running agents in parallel, they're compute constrained on those agents running ML experiments.
This is a very important point, especially due to Amdahl's law implications leading to a maximum rate of progress in line with the speed of ablation research runtime durations (which presumably aren't hard take-off level).
People keep forgetting Amdahl's law. If you fully automate a 1/X proportion of the work, you can speed up overall progress by a maximum of a factor of X — the part you couldn't automate becomes the bottleneck. Analysis from Anthropic and OpenAI have both made it clear that currently the proportion of the total tasks involved in AI capabilities research they can automate is maybe up to a half, so they're at most doubling the rate of progress. Thoe were, of course, the easy-to-automate half. AI skills are very spiky, so the easy half happens a good way before the hardest bits — likely several years before at current rates of progress, so probably less as progress accelerates.
So I think we have low-single-digit years, not single-digit months, before FOOM. Which is plenty short enough to be scary.
Another significant concern is that is that capabilities increase accelerates faster than prosaic alignment improvement, since it's easier to automate more of, and as a result we get a serious loss-of-control before the FOOM happens.
This post doesn't communicate much that is new (at least, to a Lesswrong audience). It doesn't connect known facts or ideas in ways that aren't already part of this forum's collective consciousness. It feels more like a 'doomer' post that serves to infect the reader with feelings of horror, doom, and obsolescence. I really wish I saw less of this kind of writing on LW. Its increasing presence makes me substantially less excited to check the forum. Strong downvote.
I would much prefer reading content that communicates the writer's feelings on doom as an object of discussion in itself (as opposed to it being implicitly baked in as part of a superficially object-level argument).
It's a bit arbitrary to downvote and comment on this particular post; such doomer posts are a dime a dozen on here these days. I guess I clicked on this one and read it due to my general appreciation for the OP (I usually ignore doomer posts altogether).
2
1I'm fascinated by how people make sense of a world that seems to become more unpredictable and dangerous. Recently, a young friend wondered why I chose to have children even though I'm aware of AI. In LW culture, the "feeling the AGI" meme seems prominent, sometimes almost being a signal of being an insider or taking AI seriously. I think that's a shame. It implies that there is one "correct" way of feeling about AI developments. "Things are going fast, we're losing control, therefore despair." Linking object level claims to an emotional conclusion of resignation and despair seems fundamentally off in important ways. This post does just that. I'm a proponent of the attitude Viktor Frankl describes in his "Man's Search for Meaning". It (almost) doesn't matter how dire circumstances are. As a matter of subjective experience, there are many ways to make meaning of them.
Ashe's comment resonates with me. While reading the post, I indeed experienced a sense of dread. I don't endorse this, however. Object level discourse is useful. Object level discourse implicitly entangled with personal conclusions, however, can end up being a Trojan horse. It can infect readers with particular emotions or narrow interpretations of reality. We should be aware of that.
My goal here is not to invalidate OP's personal interpretation, whatever it may be. I merely want to present one alternative. Taking AI seriously does not imply having to despair. Personally, I want to build a fulfilling life, regardless of the turns it might take soon or how long it ends up being. I want to cultivate meaningful relationships with my friends and family. I want to find the courage to do the right thing and the determination to persevere in the face of obstacles. I want to have a clear-eyed view of the world. When what I'm seeing scares me, I want to turn towards equanimity or hope.
This post doesn't communicate much that is new
Sure there's no new ideas, but clearly it's very important if the presented timeline is accurate?
Effective long-horizon continual learning is likely the last remaining step to superintelligence. It could be a harder problem than a Millennium Prize problem; I consider that unlikely.
Why do you think this? Cf. https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#_We_just_need_X__intuitions
FWIW, I mostly agree with OP that "effective long-horizon continual learning is likely the last remaining step to superintelligence": there are at least a half-dozen widespread/well-known suggestions on what X might be, and for most of these I personally don't think it's likely to delay us long once the labs put their minds and funds to it. But this one we actually know is conceptually hard, because continual learning in human brains is a Heath-Robinson architectural contraption with at least half-a-dozen separate modules bolted together, people have been working on this for many years, and while we've made some progress, we're still a long way from as-good-as-a-human.
If this timeline is correct, what do you think people should do?
I personally think the timeline is underestimated. This strikes me as a "me too, with bells on" piece.
Even within mathematics, what AI has been doing recently has been somewhat limited: it finds counterexamples a lot better than it finds proofs, makes interesting conjectures, or builds understanding of an area. The recent spectacular results have almost all been counterexamples to well-known long-standing conjectures, based on exhaustively applying techniques or ideas developed by others. Which is not nothing, but it's also not all of mathematics.
> AI is capable of revolutionary advances in mathematics. Machine learning research is not different in kind.
> Internal models are significantly ahead of released ones
I think those are wrong. E. g., here roon says that models are still bad at DL research and that he doesn't think Anthropic has internal models significantly ahead of the frontier, which presumably also reflects the situation at OpenAI. This may not be entirely accurate and complete information, but I don't think he's outright lying or is even intentionally misleading about this.
Effective long-horizon continual learning is likely the last remaining step to superintelligence. It could be a harder problem than a Millennium Prize problem; I consider that unlikely.
For the purposes of effective forecasting, I don't recommend collapsing problem difficulty to a one-dimensional axis. (Is playing Go as well as AlphaZero harder or easier than resolving a Millennium Problem? I'd say it's definitely harder in certain senses. And yet here we still are.)
The pace of progress being continuously underestimated is a good point. But it could also be that mathematics research is an especially verifiable case that has historically received far less effort overall than ML research
Intuitively for me, this probability is probably not 0% but it's low. If it does happen my money is on a primarily taste-driven explosion. Basically the AI gains superhuman taste on less-verifiable parts of ML research and is now able to design better and more ingenious algorithms and training methods than any human ever thought of. (Rather than the humans coming up with ideas, the AIs would now come up with their own high-level creative ideas which are superior to the human ones.)
It would also have good enough project management taste to effectively lead the whole project end to end. That way the algorithmic efficiency gains might compound dramatically quickly with the existing amount of compute available at frontier AI labs, and the main bottleneck being sequential agent run time and training time as they design and test ever-better methods and replace themselves with the next model in a loop.
Whether we get "superintelligence by Christmas" or "AI 2027" in that scenario would then depend on by how much the AIs can exceed human judgement and how much algorithmic gains each increased step in judgement contributes to even more increases in judgement (multiplied by training and development speed the AIs can achieve).
Effective long-horizon continual learning is likely the last remaining step to superintelligence. It could be a harder problem than a Millennium Prize problem; I consider that unlikely. [3]
I would suspect labs have spent enormous amounts of capital and time trying to solve this problem, since from their perspective solving it = "winning" the race, and so far have failed. At the same time OAI spent a few tens of millions on RL runs + tokens to solve millennium prize problems and succeeded.
Generally RL works in highly verifiable domains (like math) where updates are easy based on measurable progress to success, and poorly in areas where there isn't a clear hill to climb towards success (such as research taste).
My base expectation is to see continued rapid progress (up to the point of great danger) in verifiable domains over the next 12-18 months, and extremely slow and uneven progress across unverifiable domains (if any). I do wonder if there isn't a trade off between large scale RLVF and "taste" in the sense that growing minds to solve extremely well specified problems over long horizons may reduce their ability to explore different possibility spaces.
I note they don't have AL5 on the graph yet; I think that would need to appear at least a week or two before fully automated RSI. I wonder if they'll keep the dashboard up to date daily.
FOOM should probably should be your *default expectation*.
The idea of "Foom" came about at least one paradigm ago, when RL alone was what people thought was our best bet at making superintelligence. A model gets a sufficiently good training environment (this assumption did a lot of work) that involved altering its own weights and architecture, maybe through something like weight surgery, and managed to bootstrap itself to better performance very quickly.
LLMs are a different animal, and there are good reasons to be skeptical of a vertical shift in capabilities, even if you end up believing one will happen anyways. For one, they're initialized from a wide swath of human text, and, so far, what we've done has generally involved re-eliciting human capabilities that were somewhat latent within the base model already, and what most people would recognize as superintelligence without argument may very well be a lot harder to obtain than the performance of an increasingly narrow subset of human writers. Secondly, LLM training runs are extraordinarily expensive, such that most companies seem to reserve most of their compute for training and research. A lot of the bottleneck on progress seems to be hardware, and there's no getting around the physical challenges involved in manufacturing a new generation of hardware.
To be clear, LLMs can already do a lot of world-changing things, and even if they hit a previously-undiscovered hard cap on capabilities tomorrow, we're going to have a decade or more of institutions catching up, often in terrifying ways, with surveillance, censorship, and unemployment all immediate concerns. But everything I know about the architecture, and every paper I've read on model progress, training, and performance, all point to "Foom" not being the default expectation.
Upliftingly cheery take as always Alex
I think it is plausible a strong form of recursive self-improvement[1] is imminent or already underway, and that we may be on track for superintelligence by Christmas of this year if racing continues.
This is substantially faster than any forecast, including ones like AI 2027 that were considered outrageously fast a year ago. It is faster than I myself expected even a week ago. I don't work at a scaling lab. I don't know more than is public knowledge.
Let me be perfectly clear: what I am saying is absolutely nuts. Extraordinary claims require extraordinary evidence. I claim we have now received said evidence and you should update accordingly.
FOOM should probably should be your *default expectation*.
People have strong status quo bias. Your default expectation should be that things will radically speed up.
We are not at the ceiling of intelligence.
We should probably expect the transition to superintelligence to be incredibly fast. [2]
RSI is a positive feedback loop, so it is inherently (hyper)exponential. Everything is an S-curve eventually, but nothing suggests the ceiling is anywhere near human level, or that it happens at a human timescale.
AI is capable of revolutionary advances in mathematics. Machine learning research is not different in kind.
Navier-Stokes was resolved with a counterexample, which is generically easier than a positive resolution. OpenAI has told the press it has substantial progress on a second Millennium Prize problem. Rumours name the Hodge conjecture for OpenAI [ and Birch–Swinnerton-Dyer for Anthropic. It is difficult to overstate just how incredibly hard these problems are.
Machine learning engineering is not substantially different from mathematics: fundamentally AI lives on computers. There is no or little 'real-world friction'.
A lot is made of progress being confined to domains with verifiable objectives. So the argument goes: AI can solve well-posed problems with a checker, and most of the world has no checker. There is some truth to this but it overstates the case.
These mathematical problems (Millenium prize problems, FrontierMath4) are well-defined but they have no clear gradient: you don't get 30% of a proof for 30% of the work. So this was long-horizon search without a dense reward. Regardless, progress in AI has overwhelmingly come from simple hillclimbing and picking low-hanging fruit.
Effective long-horizon continual learning is likely the last remaining step to superintelligence. It could be a harder problem than a Millennium Prize problem; I consider that unlikely. [3]
The speed of AI progress continues to be underestimated; by superforecasters and even by the researchers themselves.
Even after two years of enormous progress and a lot of updating ('feeling the AGI'), forecasts have still trailed what actually happened. You Should Update On This.
Navier-Stokes and FrontierMath Tier 4 fell substantially ahead of every schedule I know of (numbers below). This should update you toward hyperexponential FOOM scenarios, not 'slow' takeoff.
Actual: 40% at end of 2025, 94% in September 2026; Navier Stokes Millenium prize problem settled last month, with rumors that two more are solved as well. Even the developers of the technology itself are surprised by the speed - Noam Brown, the RL lead of OpenAI says he was taken aback at the pace of progress and hesitates to forecast beyond three months.
Internal models are significantly ahead of released ones;
Intuitions from working directly with publicly available models is may be misleading about the pace of progress. The Navier-Stokes model, which started training on 28 August, is said to be "significantly more capable" than GPT-6 Astra. Anthropic has a model "somewhat more capable" than Mythos 5 it does not plan to release. Claude writes a large majority of the code merged into production. OpenAI's own report of 6 September says its top 10% of researchers now spend $7,000 a day on tokens. That is about $2.5M a year per researcher, against a median new-hire salary of about $1.5M. The median researcher spends $600 a day, up from $162 in July. Agent labour-hours passed human labour-hours inside OpenAI in June and stood at 3.14x by mid-August.
Intuitions about timing from pre-training runs are misleading since most progress comes from RL, unhobbling and algorithmic innovations
In the previous paradigm the pre-trained models were already more than enough; unhobbling and RL was what was left.
One such unhobbling is cooperating agent swarms.
Enter the Swarm
OpenAI and Anthropic seem to have discovered how to use many instances of the same model ['agents'] together much more effectively. This massively multiplies the effective brainpower that can be brought to bear on one problem. 10,000 × 88 hours is 880,000 agent-hours. Navier-Stokes was solved by a swarm of 10,000 agents. As a rule of the thumb firm productivity grows roughly with the square root of the number of employees so my guess is that this buys something like 100x over a single model.
Anthropic's own report states it has 30,000 agents running concurrently, and Claude has completely taken over 26% of all R&D.
Linear extrapolation gives complete RSI summer of 2027. It's beginning to look a lot like FOOM.
There are several interpretations of RSI. The weak version, AI doing most of the coding, is already happening. The strong version would be fundamentally new advances, not just scaling of previous approaches. Eg Strawberry/o1 & reasoning models or Astra's neuralese.
In Hanson's model of growth modes [animal brains, humans, farming, industry] each mode grew about 100x faster than the one before, and each transition took a small fraction of the previous mode's doubling time. On his numbers the next mode doubles every week or two and the transition takes a few years at most. Note that while we have seen a somewhat continuous takeoff so far; that does not preclude FOOM.
Astra reasoning in pure neuralese, with no chain-of-thought, looks like a moderate performance improvement, slightly ahead of cautious trend extrapolation. That may understate it: neuralese is plausibly the key unblocker for effective continual learning.