Anthropic's IPO roadshow video has a graph. It shows four models; I think dot 1 = Claude 3.5, dot 2 = Claude 4, dot 3 = Mythos, arrowhead = unreleased model.

The graph provided doesn't have any numbers on the Y axis. In this post I'll explain why I think the Y axis lines most likely represent a 10x jump each, and so the point at the end of the graph represents about a 30000x CM (i.e. Anthropic can currently pretrain models at the same quality with 1/30000th of the compute it needed in 2024). I'm pretty unsure about this though and it could plausibly be much higher or much lower. Keep in mind that this graph is only about pretraining and doesn't say anything about post-training.^[1]

Without extra context, it's hard to say how much the compute has actually been multiplied: if each Y gridline is 1.1x the total multiplier would be about 1.53x; if each Y gridline is 10x the total multiplier would be about 30000x. (It's not clear to me why the last point is an arrowhead instead of a dot - maybe it's supposed to be their guess of what the compute multiplier will end up at for their latest pretraining run?)

What is a CM?

From a deposition, a pretraining researcher explains what CMs are a bit better than the roadshow video:

Q. What do you mean by "makes the model dumber"?

A. Um, so the way that we normally define this is we -- because there's a notion of the compute efficient frontier. We're given a fixed budget of compute. There's an optimal way to train your model and an optimal loss you'll achieve on your objective function.

Um, so we can use this as a measuring stick comparing various models. So, for example, um, if you made the model much dumber, it might shift it down so as if you only had half as much compute to train that model.

Working backwards to estimate the actual compute multipliers

If we can estimate the CM from Claude 3.5 to Claude 4, we can work out where the rest of the points on the graph should be based on that. Here I try to estimate that number very roughly.

Estimating Claude 1.0 -> Opus 3 CM

Claude 1 was pretrained with 1.6e25 FLOPs. We don't know how many FLOPs Claude 3 had, but it probably had around the same number of parameters as Claude 1 ("Lark [Claude 1] is actually too big, so Teal [Claude 3 Opus] will likely not be that much larger") but with much more data and a better architecture.

This graph^[4] from an internal presentation shows that the actual amount of compute used to train Claude 1.0 was about 1.6e25 FLOPs. Extrapolating that graph to when Opus 3 pretraining finished would give 7.4e25 FLOPs. For various kinda complicated reasons^[5] I think this is too high and it's actually more like ~4e25 FLOPs. So Opus 3 had ~2.5x the pretraining compute as Claude 1.0.

To calculate the 1.0 -> Opus 3 CM correctly we'd have to know how good 1.0 would be if we scaled it up to use as many FLOPs as Opus 3. We don't know that, but if you extrapolate based on how Llama3 scaled, you get that it would take 13x to 32x the compute. (Various other measures I tried roughly lined up with that too.) In other words, Opus 3 was as good as scaling up Claude 1.0 13x to 32x, but only took 2.5x the compute.

So 20 / 2.5 = 8x CM in the year between Claude 1.0 and Opus 3. That's an average of 1.005713 CM per day, so if we assume the rate of progress was the same for the 8 months between Claude 3.5 and 4 that's a 4x CM.

Estimating from individual CMs

There are multiple sources of pretraining compute multipliers: better pretraining data, a better training process that requires less tokens for the same capabilities, and better architectures.

For data quality related CMs:

• "MMLU is too noisy to measure anything less than 1.1CM"
• A 1.5x CM is insanely high (and maybe wrong?)
• A 2x CM is so high it's assumed that there must have been accidental training on test data

So a normalish CM is maybe around 1.2? And maybe there'd be a few of them between model releases? Maybe we'll have ~6 ~1.2x CMs or ~4 ~1.3x CMs, for a ~3x CM between Claude 3.5 and Claude 4. I'm not sure if that's reasonable?

Working out the rest of the data points

If we assume a 3x CM for Claude 3.5 to Claude 4, then the grid lines are about 10x apart. 3x seems roughly reasonable given my estimates above, and it would make sense for the gridlines to be 10x apart each. Here's a widget Claude made where you can try different values for that first step, and see what that implies.

If you use try a CM of 5x for the first step it implies a total CM of ~3.3M and 10x would imply ~2.1B. Those would be absurdly and unreasonably large, so the first step CM is probably in the 3-4 region.

What the graph means

Keep in mind that you can improve pretraining performance on code just by changing the data mix (swapping out non-code documents for code documents). That would improve performance on code to the detriment of non-code. Anthropic has always had a lot of focus on making models good for coding, but I think it's fair to assume that they've been tilting the data mix more towards code since Claude 3.7 Sonnet (released February 2025) which was released alongside Claude Code. I doubt this skews the graph too much though.

Conclusion

Let me know if you have any questions! I'd love it if Anthropic was more transparent here; it seems valuable for them to communicate about the rate of progress in AI development.

1. ^{^}

   This makes it less useful for understanding RSI dynamics, since task-based RL is very important for model performance at model R&D.

2. ^{^}

   This transcript strongly implies that the only reason it was stopped was because Anthropic's lawyers advised that it should be stopped (presumably due to copyright concerns).

3. ^{^}

   This seems pretty clear to me, some evidence is that Claude 3 models use a different tokenizer than earlier ones.

4. ^{^}

   I think that graph is just pretrain compute, but it's not entirely clear to me. This doesn't matter too much though, since we only actually need the number for lark which I guess probably had >99% of compute spent on pretraining.

5. ^{^}

   I might explain this in a later post