it can be deeply emotionally painful to part ways with deeply held beliefs
This is not necessarily the case, not for everyone. Theories and their credences don't need to be cherished to be developed, or acted upon, they only need to be taken seriously. Plausibly this can be mitigated by keeping identity small, accepting only more legible things in the role of "beliefs" that can have this sort of psychological effect (so that they can be defeated through argument alone). Legible ideas cover a surprising amount of territory, there is no pragmatic need to treat anything else as "beliefs" in this sense, all the other things can remain ambient epistemic content detached from who you are. When more nebulous worldviews become part of one's identity, they become nearly impossible to dislodge (and possibly painful, with enough context and effort). They are still worth developing towards eventual legibility, and not practical to argue with (or properly explain).
Thus arguing legible beliefs should by their nature be less intrusive than arguing nebulous worldviews. And perhaps nebulous worldviews should be argued against being held as "beliefs" in the emotional sense in general, regardless of their apparent correctness, as a matter of epistemic hygiene. Ensuring by habit you are not going to be in the position where you have "beliefs" that would be painful to part ways with, and also can't be pinned down clearly enough to dispel.
I get a sense "RSI" will start being used to mean continual learning or even just memory features in 2026, similarly to how there are currently attempts to dilute "ASI" to mean merely robust above-human-level competence. Thus recursively self-improving personal superintelligence becomes a normal technology through the power of framing. Communication can fail until the trees start boiling the oceans, when it becomes a matter of framing and ideology rather than isolated terminological disputes. That nothing ever changes is a well-established worldview, and it's learning to talk about AI.
The end states of AI danger need terms to describe them. RSI proper is qualitative self-improvement, at least software-only singularity rather than merely learning from the current situation, automated training of new skills, keeping track of grocery preferences. And ASI proper is being qualitatively more capable than humanity, rather than a somewhat stronger cognitive peer with AI advantages, technology that takes everyone's jobs.
The crux is AIs capable at around human level, aligned in the way humans are aligned. If prosaic alignment only works for insufficiently capable AIs (not capable of RSI or scalable oversight), and breaks down for sufficiently capable AIs, then prosaic alignment doesn't help (with navigating RSI or scalable oversight). As AIs get more capable and still can get aligned with contemporary methods, the hypothesis that this won't work weakens. Maybe it does work.
There are many problems even with prosaically aligned human level AIs, plausibly lethal enough on their own, but that is a distinction that importantly changes what kinds of further plans have a chance to do anything. So the observations worth updating on are not just that prosaic alignment keeps working, but that it keeps working for increasingly capable AIs, closer to being relevant for helping humanity do its alignment homework.
Plausibly AIs are insufficiently capable yet to give any evidence on this, and it'll remain too early to tell all the way until it's too late to make any use of the update. Maybe Anthropic's RSP could be thought of as sketching a policy for responding to such observations, when AIs become capable enough for meaningful updates on feasibility of scalable oversight to become accessible, hitting the brakes safely and responsibly a few centimeters from the edge of a cliff.
2
1
1There are many missing cognitive faculties, whose absence can plausibly be compensated with scale and other AI advantages. We haven't yet run out of scale, though in 2030s we will (absent AGI, trillions of dollars in revenue).
The currently visible crucial things that are missing are sample efficiency (research taste) and continual learning, with many almost-ready techniques to help with the latter. Sholto Douglas of Anthropic claimed a few days ago that probably continual learning gets solved in a satisfying way in 2026 (at 38:29 in the podcast). Dario Amodei previously discussed how even in-context learning might confer the benefits of continual learning with further scaling and sufficiently long contexts (at 13:17 in the podcast). Dwarkesh Patel says there are rumors Sutskever's SSI is working on test time training (at 39:25 in the podcast). Thinking Machines published work on better LoRA, which in some form seems crucial to making continual learning via weight updates practical for individual model instances. This indirectly suggests OpenAI would also have a current major project around continual learning.
The recent success of RLVR suggests that any given sufficiently narrow mode of activity (such as doing well on a given kind of benchmarks) can now be automated. This plausibly applies to RLVR itself, which might be used by AIs to "manually" add new skills to themselves, after RLVR was used to teach AIs to apply RLVR to themselves, in the schleppy way that AI researchers currently do to get them better at benchmarkable activities. AI instances doing this automatically for the situation (goal, source of tasks, job) where they find themselves covers a lot of what continual learning is supposed to do. Sholto Douglas again (at 1:00:54 in a recent podcast):
So far the evidence indicates that our current methods haven't yet found a problem domain that isn't tractable with sufficient effort.
So it's not completely clear that there are any non-obvious obstacles still remaining. Missing research taste might get paved over with sufficient scale of effort, once continual learning ensures there is some sustained progress at all when AIs are let loose to self-improve. To know that some obstacles are real, the field first needs to run out of scaling and have a few years to apply RLVR (develop RL environments) to automate all the obvious things that might help AIs "manually" compensate for the missing faculties.
1Some observations (not particularly constructive):
- Training compute is relevant for the most compute-hungry models, because there it can be a taut constraint (and even then only when there would be inference hardware to serve the model once it's trained, which isn't always the case). For smaller and catch-up models, other things become more relevant as constraints, and even the relevant compute needed to make them well is no longer training compute, but research compute, or the compute that went into earlier or larger models that contributed to making the smaller model possible.
- Inference cost/speed doesn't have this issue for the smaller models, it remains both relevant and legible. Research compute and compute for implicit predecessor dependencies needed to develop smaller models are still relevant for the reproduce-much-cheaper hypothetical, but it's far less legible.
- Measuring parity based on benchmarks is suspect (even as pragmatically it's hard to use anything else), a big confused pre-RLVR model and a small competition-minded post-RLVR model will be doing things differently.
I'm responding to the claim that training scaling laws "have ended", even as the question of "the bubble" might be relevant context. The claim isn't very specific, and useful ways of making it specific seem to make it false, either in itself or in the implication that the observations so far have something to say in support of the claim.
The scaling laws don't depend on how much compute we'll be throwing at training or when, they predict how perplexity depends on the amount of compute. For scaling laws in this sense to become false, we'd need to show that perplexity starts depending on compute in some different way (with more compute). Not having enough compute doesn't disprove that the scaling laws are OK. Even not having enough data doesn't disprove this.
For practical purposes, scaling laws could be said to fail once they can no longer be exploited for making models better. As I outlined, there's going to be significantly more compute soon (this is still the case with "a bubble", which might have the power to get compute as much as 3x lower than the more optimistic 200x-400x projection for models by 2031, compared to the currently deployed models). The text data is plausibly in some trouble even for training with 2026 compute, and likely in a lot of trouble for training with 2028-2030 compute. But this hasn't happened yet, so the claim of scaling laws "having ended", past tense, would still be false in this sense. Instead, there would be a prediction that the scaling laws would in some practical sense end in a few years, before compute stops scaling even at pre-AGI funding levels. But also, the data efficiency I'm using for predicting that text data will be insufficient (even with repetition) is a product of the public pre-LLM-secrecy research that almost always took unlimited data for granted, so it's possible that spending a few years explicitly searching for ways to overcome data scarcity will let AI companies find a way to sidestep this issue, at least until 2030. Thus I wouldn't even predict that text data will run out by 2030 with a high degree of certainty, it's merely my baseline expectation.
It's still premised on the idea that more training/inference/ressources will result in qualitative improvements.
I said nothing about qualitative improvements. Sufficiently good inference hardware makes it cheap to make models a lot bigger, so if there is some visible benefit at all, this will be happening at the pace of the buildouts of better inference hardware. But also conversely, if there's not enough inference hardware, you physically can't serve something as a frontier model (for a large user base) even if that offers qualitative improvements, unless you restrict demand (with very high prices or rate limits).
So your scenario is possible; I had similar expectations a few years ago. But I'm seeing more and more evidence against it, so I'm giving it a lower probability (maybe 20%).
This is not very specific, similarly to the claim about training scaling laws "having ended". Even with "a bubble" (that bursts before 2031), some AI companies (like Google) might survive in an OK shape. These companies will also have their pick of the wreckage of the other AI companies, including both researchers and the almost-ready datacenter sites, which they can use to make their own efforts stronger. The range of scenarios I outlined only needs 2-4 GW of training compute by 2030 for at least one AI company (in addition to 2-4 GW of inference compute), which revenues of $40-80bn should be sufficient to cover (especially as the quality of inference hardware stops being a bottleneck, so that even older hardware will again become useful for serving current frontier models). Google has been spending this kind of money on datacenter capex as a matter of course for many years now.
OpenAI is projecting about $20bn of revenue in their current state, when the 800M+ free users are not being monetized (which is likely to change). These numbers can plausibly grow to at least give $50bn per year to the leading model company by 2030 (even if it's not OpenAI), this seems like a very conservative estimate. It doesn't depend on qualitative improvement in LLMs or promises for more than a trillion dollars in datacenter capex. Also, the capex numbers might even scale down gracefully if $50bn per year from one company by 2030 turns out to be all that's actually available.
Since the end of (very weak) training scaling laws
Precisely because the scaling laws are somewhat weak, there was nothing so far to indicate they are ending (the only sense in which they might be ending is running out of text data, but models trained on 2024 compute should still have more than enough). The scaling laws held for many orders of magnitude, they are going to hold for a bit further. It's plausibly not enough, even with something to serve the role of continual learning (beyond in-context learning on ever larger contexts). But there is still another 100x-400x in compute to go, compared to the best models deployed today. Likely the 100x-400x models will be trained in 2029-2031, at which point the pre-AGI funding for training systems mostly plateaus. This is (a bit more than) a full step of GPT-2 to GPT-3, or GPT-3 to original Mar 2023 GPT-4 (after original Mar 2023 GPT-4 and with the exception of GPT-4.5, OpenAI's naming convention no longer tracks pretraining compute). And we still didn't see such a full step compared to original Mar 2023 GPT-4, only half of a step (10x-25x), out of the total of 3-4 halves-of-a-step (2022-2030 training compute ramp, 2000x-10,000x in total, at higher end if BF16 to NVFP4 transition is included, at lower end if even in 2030 there are no 5 GW training systems and somehow BF16 needs to be used for the largest models).
Since original Mar 2023 GPT-4, models that were allowed to get notably larger and made full use of the other contemporary techniques only appeared in late 2025 (likely Gemini 3 Pro and Opus 4.5). These models are probably sized compute optimally for 2024 levels of pretraining compute (as in 100K H100s, 10x-25x the FLOPs of original Mar 2023 GPT-4), might have been pretrained with that amount of compute or a bit more, plus pretraining scale RLVR. All the other models we've seen so far are either smaller than compute optimal for even 2024 levels of pretrained compute (Gemini 2.5 Pro, Grok 4, especially GPT-5), or didn't get the full benefit of RLVR compared to pretraining (Opus 4.0, GPT-4.5) and so in some ways looked underwhelming compared to the other (smaller) models that were more comprehensively trained.
The buildout of GB200/GB300 NVL72 will be complete at flagship model scale in 2026, and makes it possible to easily serve models sized compute optimally for 2024 levels of compute (MoE models with many trillions of total params). More training compute is currently available and will be available in 2026 than what was there in 2024, but for most of the inference hardware currently available it won't be efficient to serve models sized compute optimally for this compute (at tens of trillions of total params), except with Ironwood TPUs (which are being built in 2026, for Google and Anthropic) and then Nvidia Rubin Ultra NVL576 (which will only get built in sufficient amounts in 2029, maybe late 2028).
So the next step of scaling will probably come in late 2026 to early 2027 from Google and Anthropic (while OpenAI will only be catching up to late 2025 models from Google and Anthropic, though of course in 2026 they'll have better methods than Google and Anthropic had in 2025). And then training compute will still continue increasing somewhat quickly for models until 2029-2031 (with 5 GW training systems, which is at least $50bn per year in training compute, or $100bn per year in total for each AI company if inference is consuming half of the budget). After Rubin Ultra NVL576 (in 2029) and to some extent even Ironwood (in 2026), inference hardware will no longer be a notable constraint on scaling, and after AI companies are working with 10 GW of compute (half for training, half for inference), pretraining compute will no longer be growing much faster than price-performance of hardware, which is much slower than the buildout trend of 2022-2026, and even than the likely ramp-off in 2026-2030. I only expect 2 GW training systems in 2028, rather than the 5 GW that the 2022-2026 trend would ask for in 2028. But by 2030 the combination of continuing buildout and somewhat better hardware should still reach the levels of what would be on-trend for 2028, following 2022-2026.
Even for danger that comes from superhumanly and robustly competent AIs, these AIs might've been to a significant extent created by idiosyncratically flawed AIs of jagged competence. The flaws of these predecessor AIs then shape the danger of their more capable successors, making these flaws a point of intervention worth addressing, even when the AIs with these flaws are not very dangerous directly. Similarly to how humanity is not dangerous directly to a superintelligence, except in how humanity would be able to create another superintelligence if left unchecked.
does not seem like it serves any defensible goal
That shouldn't be a reason not to do a thing (by itself, all else equal).
Legible ideas (that are practical to meaningfully argue about) cover a lot of ground, they are not as hazardous as part of identity. And less well-defined but useful/promising/interesting understandings don't need to become part of identity to be taken seriously and developed. That's the failure mode at the other extreme, when anything insufficiently scientific/empirical/legible/etc. gets thrown out with the bathwater.
Probably when something is easy to defeat (admits argument, legible), it's not that painful to let it go. The pain is the nebulous attachment fighting for influence, that it won't be fully defeated even when you end up consciously endorsing a change of mind. Thus ideologies are somewhat infeasible to change, they'll keep their hold even long after the host disavows them. A habit of keeping such things at a distance benefits from other people not feeding their structurally hazardous placement (as emotionally load bearing pillars) with positivity. But that's distinct from viewing positively the development of even such hazardous things, handling them with appropriate caution.