Link to long comments that I want to pin, but are too long to be pinned:

https://www.lesswrong.com/posts/Zzar6BWML555xSt6Z/?commentId=aDuYa3DL48TTLPsdJ

https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/?commentId=Gcigdmuje4EacwirD

https://www.lesswrong.com/posts/DCQ8GfzCqoBzgziew/?commentId=RhTNmgZqjJpzGGAaL

https://www.lesswrong.com/posts/NjzLuhdneE3mXY8we?commentId=rcr4j9XDnHWWTBCTq

My own take re rationalization/motivated reasoning is that at the end of the day, no form of ethics can meaningfully slow it down if the person either can't credibly commit to their future selves, or simply isn't bound/want to follow ethical rules, so the motivated reasoning critique isn't EA specific, but rather shows 2 things:

1. People are more selfish than they think themselves to be, and care less about virtues, so motivated reasoning is very easy.

2. We can't credibly commit our future selves to do certain things, especially over long timeframes, and even when people do care about virtues, motivated reasoning still harms their thinking.

Motivated reasoning IMO is a pretty deep-seated problem within our own brains, and is probably unsolvable in the near term.

A core issue here that is repeated is that since AI progress has been (so far) slower than super-exponential or faster-growing functions, and is merely growing at an exponential rate as defined by time horizons, it turns out that there's a very, very large difference between acing benchmarks and actually posing enough of an existential risk to actually serve as a useful red line, and due to the jagged frontier plus progress coming from many small improvements in compute, it's a lot harder to make clear red lines or get definitional clarity.

More generally, one of the takeaways from time horizons work is that by default, there will probably be no legible clear red lines, so any warning shots need to be prepared for.

I agree with a lot of this post, but one other motivation (at least for me) is that checking how much algorithmic progress exists/comes from compute is that it's relevant to predictions around recursive self-improvement/software intelligence explosion without increasing compute, assuming you have AIs that can fully automate AI research, and more generally informs takeoff speeds, and I take the Gundleach et al paper as evidence that pure software improvements are more likely to cap out at relatively modest impacts than creating a self-sustaining feedback loop of progress, slowing down AI takeoff by quite a bit until we massively scale up compute for AGIs.

(The evidence value is reduced but not eliminated by the fact that they tested it on LLMs).

The nuance was in saying that their framework can't predict whether or not data or compute scaling made the majority of improvements, nor can they separate out data and compute improvements, but the core finding of algorithmic efficiency being almost all compute-scaling dependent still holds, so if we had a fixed stock of compute now, we would essentially have 0 improvements in AI forever.

Another possibility, in principle, for why automating AI R&D doesn't lead to an intelligence explosion is because a very large percentage of the progress (at that part of development trajectory) is driven by scaling relative to algorithmic progress.

This is actually happening today, so the real question is why algorithmic progress returns will increase once we attempt to fully automate AI R&D, rather than why we won't get an intelligence explosion.

More specifically, the algorithmic progress that has happened is basically all downstream of more compute going into AI, and algorithmic efficiency is dependent on compute scale being larger and larger to reap the gains of better algorithms.

Just LLMs with pretraining, giving superintelligent in-context learning, making AIs able to figure out how to work around their hobblings using just in-context reasoning. This clearly doesn't work at 2024 levels of compute (which only got to be observed in 2025), and after 2026 levels of compute natural text data starts running out. Maybe there are some relevant sparks at 2029-2031 levels of compute (5 GW), but probably not sufficient on its own.

I want to flag that you estimated that post-2028, we'd slow down due to the MoE data wall, and while MoE is more compute efficient, I don't think it's so much more compute efficient that people won't take the hit of less compute efficiency if data efficiency becomes more relevant post-2028, and we can probably continue mostly pre-training up until the early 2030s based on data limitations alone. That said, I agree this is likely insufficient on its own, so I'm not claiming that pure LLM AGI is a likely path.

Continual learning doesn't seem directly relevant, but maybe it helps with teaching LLMs to RLVR LLMs. It doesn't seem directly relevant because it probably doesn't directly train mildly superhuman capabilities. But since the generalizing glue of adaptation would make everything work better, it might enable method (2) to go forward where it would fail on its own. The relevant benchmarks might be the same as for (2), but only if they don't already saturate by the time there's useful continual learning (in some nontrivial sense of the term).

I think continual learning is more likely to end up important in its impact on AI company TAM. If it manages to 10x it, then it will thereby 10x the compute, though it'll proably take until 2030-2035 to actually build the kind of research/training compute (20-50 GW) that a trillion dollars of revenues buys (even if 2026 already shows that this is on track to happen).

I think this is probably true for the near term (as in 5-10 year timelines), but over a 15-20 year span with Moore's law, I'd be more confident that continual learning alone could plausibly train in mildly superhuman capabilities, for the same reasons why certain researchers IRL are way more productive than others.

This does depend on human level or better sample efficient learning to work out, though.

That said, I agree with your methodology much more than Daniel Kokotajlo's methodology on AI prediction (though I do think METR time horizons are more useful and less anchored to specific paradigms than you say it is).

It's actually worse, in that almost all such simulators are hypercomputational and have literally infinite compute, memory and data, as well as their programs being infinitely large, so it's literally useless to think/write with potential simulators as an audience.

The answer I suspect has to do with a focus on short timelines combined with more bearish views on software-only singularities than most of LW.

Short timelines means there's a lot less compute resources to run AIs because Moore's law hasn't had as much of an effect compared to future dates, combined with less compute being built overall, meaning we can only run millions or at best tens of millions of AIs.

One thing I note also is that he does think that AIs can increase their intelligence, but that this runs into steep diminishing returns early enough that other factors matter more.

Also, in practical terms, gradual disempowerment does not seem particularly convenient set of ideas for justifying that working in an AGI company on something very prosaic which helps the company is the best thing to do.

The bigger issue, as Jackson Wagner says is that there's a very real risk that will be coopted by people who want to talk mostly about present-day harms of AI, and thus at best siphoning resources from actually useful work on gradual disempowerment threats/AI x-risk in general, and at worst creating polarization around gradual disempowerment with one party supporting gradual disempowerment of humans and another party opposing gradual disempowerment, while the anti-gradual disempowerment party/people become totally ineffective at dealing with the problem because it's been taken over by omnicause dynamics.