In this post, I describe a simple model for forecasting when AI will automate AI development. It is based on the AI Futures model, but more understandable and robust, and has deliberately conservative assumptions. At current rates of compute growth and algorithmic progress, this model's median prediction is >99% automation...
This work was done while at METR. Introduction GDM recently released a paper (Emmons et al.) showing that, contrary to previous results, the chain-of-thought (CoT) of language models is more faithful when the model’s CoT is necessary for it to complete a task. They examine three settings where an actor...
Summary METR blogpost | X | Github In the paper Measuring AI Ability to Complete Long Software Tasks (Kwa & West et al. 2025), METR defined an AI model’s 50% time horizon as the length of tasks (measured by how long they take human professionals) that it can complete autonomously...
arXiv | project page | Authors: Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, Gao Huang This paper from Tsinghua find that RL on verifiable rewards (RLVR) just increases the frequency at which capabilities are sampled, rather than giving a base model new capabilities....
As Americans know, the electoral college gives disproportionate influence to swing states, which means a vote in the extremely blue state of California was basically wasted in the 2024 election, as are votes in extremely red states like Texas, Oklahoma, and Louisiana. State legislatures have the Constitutional power to assign...
Suppose you can tell your AI to meet a certain spec (e.g. cure cancer), but most plans that meet the spec are unsafe (e.g. involve killing everyone, or so Rob Bensinger thinks). In these cases, a quantilizer is insufficient for safety due to instrumental convergence.[1] But suppose we can also...
This is the appendix to the previous post on Goodhart’s Law and KL regularization, containing all of our proofs. Theorem about distributions Theorem 1: Given any heavy-tailed reference distribution Q over R with mean μQ, and any M,ϵ>0, there is a distribution P with mean μP>M and DKL(P∥Q)<ϵ. Proof: WLOG...
I think the crux vs papers like Jones and Tonetti (2026) is whether AI R&D is much easier to automate than tasks in the general economy. From a skim, they include AI increasing idea production but not specifically work at AI labs.
As for the shape of the automation curve, agree it's very important. I'd be interested in any ideas on how to measure it.
Ok, here's what I think the primary scenario as it applies to me:
The outcome-steering effect of future AIs will lose any discernable “flavor” or “personality” leaking from any details of its implementation that used to relate to why the implementation wasn’t robustly superhuman. Even attempts to probe the AI’s “mechanistically interpretable conditional behavior” will offer no predictive power beyond modeling the AI as an outcome-steering engine.
This isn't a good argument. For one thing, humans still have personalities if they could think at 1000x speed. Now it's true that Stockfish has less personality than human chess players because it's optimized for the single goal of winning a chess game. But even if we assume superintelligences have fixed goals like this, conditional behavior will still clue... (read more)
I am skeptical that uplift measurements actually give much tighter confidence intervals.
I think we might already have evidence against longer timelines (2049 timelines from doubling difficulty growth factor of 1.0). Basically, the model needs to pass a backtest against where uplift was in 2025 vs now, and if there's significant uplift now and wasn't before 2025, this implies the automation curve is steep.
Suppose uplift at the start of 2026 was 1.6x as in the AIFM's median (which would be 37.5% automation if you make my simplifying assumptions). If we also know that automation at start of 2025 was at most 20%, this means automation is increasing at 0.876 logits per year, or... (read more)
Maybe to rule out luck you could give an estimate of your average beta and Sharpe ratio, which mostly only depend on your returns over time. Also, are you planning to keep actively trading part-time?
Here's some feedback. Reading over it now it sounds overly harsh (sorry!) but I mostly endorse it. Looking forward to further improvements!
In this post, I describe a simple model for forecasting when AI will automate AI development. It is based on the AI Futures model, but more understandable and robust, and has deliberately conservative assumptions.
At current rates of compute growth and algorithmic progress, this model's median prediction is >99% automation of AI R&D in late 2032. Most simulations result in a 1000x to 10,000,000x increase in AI efficiency and 300x-3000x research output by 2035. I therefore suspect that existing trends in compute growth and automation will still produce extremely powerful AI on "medium" timelines, even if the full coding automation and superhuman research taste that drive the AIFM's "fast" timelines (superintelligence by ~mid-2031)... (read 2285 more words →)
The primary takeaway here is their demonstration that a 0.1 absolute change in this correlation—shifting from, say, 0.5 to 0.6—can have a bigger impact on the positive predictive value of one’s R&D pipeline than screening ten times, or even a hundred times, more candidates.
Mainly due to various frictions like not knowing if the different tax treatment is acceptable, and the fact that they often cost over $30,000 each. Also due to luck, my SPY options are up over 100% so I don't want to sell them until I need to roll them in ~December, and buying SPX options on top of that would leave me overexposed. This is plausibly a mistake though.
Maybe Google or Amazon should employ burglars-- not only do they have a higher legitimate revenue than Facebook, they have other advantages. Google has a better reputation than Facebook allowing them to get a higher percentage of revenue from crime before it becomes a PR issue. Amazon already has an extensive logistics network they could use for getaway vehicles.
Consider this toy game-theoretic model of AI safety: The US and China each have an equal pool of resources they can spend on capabilities and safety. Each country produces either AIs aligned to it or misaligned AIs. If one country's AI is misaligned, both payoffs are zero. If both AIs are aligned, a payoff of 100 is split proportional to the resources invested in capability.
The ratio of safety to capabilities investment determines the probability of misalignment; specifically, probability of misalignment is c / (ks + c) for some "safety effectiveness parameter" k. (At k=1 we need a 1:1 safety:capabilities ratio to have a 50% chance of alignment per AI, whereas at k=100... (read more)
I've been thinking about what retirement planning means given AGI. I previously mentioned investment ideas that, in a very capitalistic future, could allow the average person to buy galaxies. But it's also possible that property rights won't even continue into the future due to changes brought about by AGI. What will these other futures look like (supposing we don't all die) and what's the equivalent of "responsible retirement planning"?
Is it building social and political capital? Making my impact legible to future people and AIs? Something else? Is any action we take today futile?
Today I played an AI Futures tabletop exercise [1]. I'd done one before with a scenario similar to Plan A, where the US executive is much more doomy and competent than we expect and leads an international agreement to pause AI for as long as possible. This time, we played "Optimal China", where instead China was doomy and competent, and considered both x-risk and US AI dominance unacceptable. All other players (frontier labs, safety community, Europe) basically role-play the way we expect them to act, and the AI player draws the AI's goals from a distribution that includes misaligned goals, aligned goals, and combinations thereof.
In Plan A... (read 517 more words →)
If faraway galaxies are ever auctioned off, you might think they'd be exorbitantly expensive. However, I think the market price of an average galaxy would only be equivalent to between $250 and $25,000 invested in the stock market today. (epistemic status: my modal guess, pretty weakly held)
Assumptions:
In Tom Davidson's semi-endogenous growth model, whether we get a software-only singularity boils down to whether r > 1, where r is a parameter in the model [1]. How far we are from takeoff is mostly determined by the AI R&D speedup current AIs provide. Because both parameters are rather difficult to estimate, I believe we can't rule out that
(This post is now live on the METR website in a slightly edited form)
In the 9 months since the METR time horizon paper (during which AI time horizons have increased by ~6x), it’s generated lots of attention as well as various criticism on LW and elsewhere. As one of the main authors, I think much of the criticism is a valid response to misinterpretations, and want to list my beliefs about limitations of our methodology and time horizon more broadly. This is not a complete list, but rather whatever I thought of in a few hours.
Most people should buy long-dated call options:
If you're early career, have a stable job, and have more than ~3 months of savings but not enough to retire, then lifecycle investing already recommends investing very aggressively with leverage (e.g. 2x the S&P 500). This is not speculation, it decreases risk due to diversifying over time. The idea is as a 30yo most of your wealth is still in your future wages, which are only weakly correlated with the stock market, so 2x leverage on your relatively small savings now might still mean under 1x leverage on your effective lifetime portfolio.
In 2026, most of your long-term financial risk comes from your job being automated,... (read 525 more words →)
Diminishing returns in the NanoGPT speedrun:
To determine whether we're heading for a software intelligence explosion, one key variable is how much harder algorithmic improvement gets over time. Luckily someone made the NanoGPT speedrun, a repo where people try to minimize the amount of time on 8x H100s required to train GPT-2 124M down to 3.28 loss. The record has improved from 45 minutes in mid-2024 down to 1.92 minutes today, a 23.5x speedup. This does not give the whole picture-- the bulk of my uncertainty is in other variables-- but given this is existing data it's worth looking at.
I only spent a couple of hours looking at the data [3], but there... (read 556 more words →)
An indiscriminate space weapon against low earth orbit satellites is feasible for any spacefaring nation. Recent rumors claim that Russia is already developing one, so I'm writing a post about it. I will explain why I think
Will nuclear ICBMs in their current form be obsolete soon? Here's the argument:
This work was done while at METR.
GDM recently released a paper (Emmons et al.) showing that, contrary to previous results, the chain-of-thought (CoT) of language models is more faithful when the model’s CoT is necessary for it to complete a task. They examine three settings where an actor model attempts to fool monitors on various tasks.
In Section 5 of the paper, the authors task an actor model with solving a difficult GPQA problem within one of three subject areas (Biology/Chemistry/Physics) while obfuscating its reasoning so that it appears to be about a different subject. A monitor model, given the answer of the actor model, is instructed to determine how confident it is that there is reasoning... (read 1459 more words →)
METR blogpost | X | Github
In the paper Measuring AI Ability to Complete Long Software Tasks (Kwa & West et al. 2025), METR defined an AI model’s 50% time horizon as the length of tasks (measured by how long they take human professionals) that it can complete autonomously with 50% probability. We estimated the time horizon of frontier models released since 2019 on a benchmark combining three sets of software and research tasks ranging from 1 second to 16 hours in length-for-humans (HCAST, RE-Bench, and SWAA, henceforth METR-HRS). METR found that the time horizon has doubled every 7 months, possibly accelerating to every 4 months in 2024.
One important limitation was the task... (read 3976 more words →)
arXiv | project page | Authors: Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, Gao Huang
This paper from Tsinghua find that RL on verifiable rewards (RLVR) just increases the frequency at which capabilities are sampled, rather than giving a base model new capabilities. To do this, they compare pass@k scores between a base model and an RLed model. Recall that pass@k is the percentage of questions a model can solve at least once given k attempts at each question.
Main result: On a math benchmark, an RLed model (yellow) has much better raw score / pass@1 than the base model (black), but lower pass@256! The authors say... (read 483 more words →)
As Americans know, the electoral college gives disproportionate influence to swing states, which means a vote in the extremely blue state of California was basically wasted in the 2024 election, as are votes in extremely red states like Texas, Oklahoma, and Louisiana. State legislatures have the Constitutional power to assign their state's electoral votes. So why don't the four states sign a compact to assign all their electoral votes in 2028 and future presidential elections to the winner of the aggregate popular vote in those four states? Would this even be legal?
The population of CA is 39.0M (54 electoral votes), and the population of the three red states is 38.6M (55 electoral... (read 231 more words →)
Suppose you can tell your AI to meet a certain spec (e.g. cure cancer), but most plans that meet the spec are unsafe (e.g. involve killing everyone, or so Rob Bensinger thinks). In these cases, a quantilizer is insufficient for safety due to instrumental convergence.[1]
But suppose we can also give the agent a dispreference for unsafe actions like murder through something like RLHF. In effect it has unsafe long-term goals but we control its immediate preferences.[2] When can we get the agent to cure cancer safely rather than murder you? Let's make a model with some unrealistic assumptions.
This is basically the simplest game in which an undesired power-increasing action becomes... (read 533 more words →)
This is the appendix to the previous post on Goodhart’s Law and KL regularization, containing all of our proofs.
Theorem 1: Given any heavy-tailed reference distribution over with mean , and any , there is a distribution with mean and .
Proof: WLOG let . We construct a sequence of distributions such that for any constant , and . We define for any thusly. Writing for the CDF and for , we let
Intuitively, we rescale the part of the distribution to the right of evenly to have total probability , which is less than 1 because .
We must check that . We can write
... (read 1666 more words →)TLDR: In the last two posts, we showed that optimizing for a proxy can fail to increase true utility, but only when the error is heavy-tailed. We now show that this also happens in RLHF with a KL penalty.
This post builds on our earlier result with a more realistic setting and assumptions:
This came up as a tangent from this question, which is itself a tangent from a discussion on The Hidden Complexity of Wishes.
Suppose we have a perfect cubical box of length 1 meter containing exactly 1 mol of argon gas at room temperature.
This came up as a tangent from @habryka and me discussing whether The Hidden Complexity of Wishes was correct.
Is the result of a US presidential election 4 years from now >0.1% contingent on quantum randomness (i.e. is an otherwise omniscient observer forecasting the 2028 election today capable of >99.9% confidence, or is there >0.1% irreducible uncertainty due to quantum mechanics observer-effects)?
I think the answer is yes, because chaotic systems will quickly amplify this randomness to change many facts about the world on election day.
I feel like you're overreacting to this. Surely the most likely explanation is that talking to LLMs is some evidence that LLMs will be aligned in the limit of power and intelligence, but (a) Davidad is overconfident, (b) this is due to the hard work of alignment researchers, and/or (c) precautionary principle, and so alignment researchers shouldn't go home just yet?