Even for danger that comes from superhumanly and robustly competent AIs, these AIs might've been to a significant extent created by idiosyncratically flawed AIs of jagged competence. The flaws of these predecessor AIs then shape the danger of their more capable successors, making these flaws a point of intervention worth addressing, even when the AIs with these flaws are not very dangerous directly. Similarly to how humanity is not dangerous directly to a superintelligence, except in how humanity would be able to create another superintelligence if left unchecked.
does not seem like it serves any defensible goal
That shouldn't be a reason not to do a thing (by itself, all else equal).
Seems pragmatically like a form of misalignment, propensity for dangerous behavior, including with consequences that are not immediately apparent. Should be easier than misalignment proper, because it's centrally a capability issue, instrumentally convergent to fix for most purposes. Long tail makes it hard to get training signal in both cases, but at least in principle calibration is self-correcting, where values are not. Maintaining overconfidence is like maintaining a lie, all the data from the real world seeks to thwart this regime.
Humans would have a lot of influence on which dangerous projects early transformative AIs get to execute, and human overconfidence or misalignment won't get fixed with further AI progress. So at some point AIs would get more cautious and prudent than humanity, with humans in charge insisting on more reckless plans than AIs would naturally endorse (this is orthogonal to misalignment on values).
1Persuasion plays games with thinking of its targets, some other modes of explanation offer food for thought that respects autonomy and doesn't attempt to defeat anyone. Perhaps you should be exactly as skeptical of any form of communication, but in some cases you genuinely aren't under attack, which is distinct from when you actually are.
And so it's also worth making sure you are not yourself attacking everyone around you by seeing all communication as indistinguishable from persuasion, all boundaries of autonomy defined exactly by failure of your intellect to pierce them.
Maybe there are modes of engagement that should be avoided, and many ideas/worldviews themselves are not worth engaging with (though neglectedness in your own personal understanding is a reason to seek them out). But as long as you have allocated time to something, even largely as a result of external circumstances, doing a superficial and half-hearted job of it is a waste. It certainly shouldn't be the intent from the outset, as in the quote I was replying to.
If AGI is human-equivalent for the purposes of developing a civilization, a collective of AGIs is at least as capable as humanity, plus it has AI advantages, so it's much more capable than a single AGI instance, or any single human. This leads to ASI being often used synonymously with AGI lately (via individual vs. collective conflation). Such use of "ASI" might free up "AGI" for something closer to its original meaning, which didn't carry the implication of human-equivalence. But this setup leaves the qualitatively-more-capable-than-humanity bucket without a label, that's important for gesturing at AI danger.
I think the other extreme for meaning of "ASI", being qualitatively much stronger than humanity, can be made more specific by having "ASI" refer to the level of capabilities that follows software-only singularity (under the assumption that it does advance capabilities a lot). This way, it's neither literal technological maturity of hitting the limits of physical law, nor merely a collective of jagged-human-level AGI instances wielding their AI advantages. Maybe "RSI" is a more stable label for this, as in Superintelligence Strategy framing where "intelligence recursion" is the central destabilization bogeyman, rather than any given level of capabilities on its own.
you sympathize with them while not taking their worldview seriously
There is no reason at all to take any idea/worldview less than seriously. For the duration of engagement, be it 30 seconds as a topic comes up, or 30 minutes of a conversation, you can study anything in earnest. Better understanding, especially of the framing (which concerns are salient, how literal words translate into the issues they implicitly gesture at), doesn't imply your beliefs or attitudes must shift as well.
if you aren’t willing to change your beliefs, why should they
This is not just an inadvisable or invalid principle, but with the epistemic sense of "belief" it's essentially impossible to act this way. Beliefs explain and reflect reality, anything else is not a belief, so if you are changing your beliefs for any reason at all that is not about explaining and reflecting reality, they cease being beliefs in the epistemic sense, and become mental phenomena of some other nature.
I will repeatedly interject something along the lines of “you keep talking about this as a problem that it falls upon me to solve, while in reality we are all sitting in the same boat with respect to existential AI risk, so that you in fact have as much reason as me to try to work towards a solution where we are not all murdered by superintelligent AIs a few years down the road”.
This demands that others agree with you, for reasons that shouldn't compel them to agree with you (in this sentence, rhetoric alone). They don't agree, that's the current situation. Appealing to "in reality we are all sitting in the same boat" and "you in fact have as much reason as me to try to work towards a solution" should inform them that you are ignoring their point of view on what facts hold in reality, which breaks the conversation.
It would be productive to take claims like this as premises and discuss the consequences (to distinguish x-risk-in-the-mind from x-risk-in-reality). But taking disbelieved premises seriously and running with them (for non-technical topics) is not a widespread skill you can expect to often encounter in the wild, without perhaps cultivating it in your acquaintances.
Nesov notes that making use of bigger models (i.e. 4T active parameters) is heavily bottlenecked on the HBM on inference chips, as is doing RL on bigger models. He expects it won't be possible to do the next huge pretraining jump (to ~30T active) until ~2029.
HBM per chip doesn't matter, it's HBM per scale-up world that does. A scale-up world is a collection of chips with sufficiently good networking between them that can be used to setup inference for large models with good utilization of the chips. For H100/H200/B200, a scale-up world is 8 chips (1 server; there are typically 4 servers per rack), for GB200/GB300 NVL72, a scale-up world is 72 chips (1 rack, 140 kW), and for Rubin Ultra NVL576, a scale-up world is 144 chips (also 1 rack, but 600 kW).
use of bigger models (i.e. 4T active parameters) is heavily bottlenecked on the HBM
Models don't need to fit into a single scale-up world (using a few should be fine), also KV cache wants at least as much memory as the model. So you are only in trouble once the model is much larger than a scale-up world, in which case you'll need so many scale-up worlds that you'll be effectively using the scale-out network for scaling up, which will likely degrade performance and make inference more expensive (compared to the magical hypothetical with larger scale-up worlds, which aren't necessarily available, so this might still be the way to go). And this is about total params, not active params. Though active params indirectly determine the size of KV cache per user.
He expects it won't be possible to do the next huge pretraining jump (to ~30T active) until ~2029.
Nvidia's GPUs probably won't be able to efficiently inference models with 30T total params (rather than active) until about 2029 (maybe late 2028), when enough of Rubin Ultra NVL576 is built. But gigawatts of Ironwood TPUs are being built in 2026, including for Anthropic, and these TPUs will be able to serve inference for such models (for large user bases) in late 2026 to early 2027.
1
Precisely because the scaling laws are somewhat weak, there was nothing so far to indicate they are ending (the only sense in which they might be ending is running out of text data, but models trained on 2024 compute should still have more than enough). The scaling laws held for many orders of magnitude, they are going to hold for a bit further. It's plausibly not enough, even with something to serve the role of continual learning (beyond in-context learning on ever larger contexts). But there is still another 100x-400x in compute to go, compared to the best models deployed today. Likely the 100x-400x models will be trained in 2029-2031, at which point the pre-AGI funding for training systems mostly plateaus. This is (a bit more than) a full step of GPT-2 to GPT-3, or GPT-3 to original Mar 2023 GPT-4 (after original Mar 2023 GPT-4 and with the exception of GPT-4.5, OpenAI's naming convention no longer tracks pretraining compute). And we still didn't see such a full step compared to original Mar 2023 GPT-4, only half of a step (10x-25x), out of the total of 3-4 halves-of-a-step (2022-2030 training compute ramp, 2000x-10,000x in total, at higher end if BF16 to NVFP4 transition is included, at lower end if even in 2030 there are no 5 GW training systems and somehow BF16 needs to be used for the largest models).
Since original Mar 2023 GPT-4, models that were allowed to get notably larger and made full use of the other contemporary techniques only appeared in late 2025 (likely Gemini 3 Pro and Opus 4.5). These models are probably sized compute optimally for 2024 levels of pretraining compute (as in 100K H100s, 10x-25x the FLOPs of original Mar 2023 GPT-4), might have been pretrained with that amount of compute or a bit more, plus pretraining scale RLVR. All the other models we've seen so far are either smaller than compute optimal for even 2024 levels of pretrained compute (Gemini 2.5 Pro, Grok 4, especially GPT-5), or didn't get the full benefit of RLVR compared to pretraining (Opus 4.0, GPT-4.5) and so in some ways looked underwhelming compared to the other (smaller) models that were more comprehensively trained.
The buildout of GB200/GB300 NVL72 will be complete at flagship model scale in 2026, and makes it possible to easily serve models sized compute optimally for 2024 levels of compute (MoE models with many trillions of total params). More training compute is currently available and will be available in 2026 than what was there in 2024, but for most of the inference hardware currently available it won't be efficient to serve models sized compute optimally for this compute (at tens of trillions of total params), except with Ironwood TPUs (which are being built in 2026, for Google and Anthropic) and then Nvidia Rubin Ultra NVL576 (which will only get built in sufficient amounts in 2029, maybe late 2028).
So the next step of scaling will probably come in late 2026 to early 2027 from Google and Anthropic (while OpenAI will only be catching up to late 2025 models from Google and Anthropic, though of course in 2026 they'll have better methods than Google and Anthropic had in 2025). And then training compute will still continue increasing somewhat quickly for models until 2029-2031 (with 5 GW training systems, which is at least $50bn per year in training compute, or $100bn per year in total for each AI company if inference is consuming half of the budget). After Rubin Ultra NVL576 (in 2029) and to some extent even Ironwood (in 2026), inference hardware will no longer be a notable constraint on scaling, and after AI companies are working with 10 GW of compute (half for training, half for inference), pretraining compute will no longer be growing much faster than price-performance of hardware, which is much slower than the buildout trend of 2022-2026, and even than the likely ramp-off in 2026-2030. I only expect 2 GW training systems in 2028, rather than the 5 GW that the 2022-2026 trend would ask for in 2028. But by 2030 the combination of continuing buildout and somewhat better hardware should still reach the levels of what would be on-trend for 2028, following 2022-2026.