GPT-3 is about 2e11 parameters and uses about 4 flops per parameter per token, so about 1e12 flops per token.

If a human writes at 1 token per second, then you should be comparing 1e12 flops to the cost per second. I think you are implicitly comparing to the cost for a ~1000 token context?

I think 1e14 to 1e15 flops is a plausible estimate for the productive computation done by a human brain in a second, which is about 2-3 orders of magnitude beyond GPT-3.

I think this is not really a coincidence. GPT-3 is notable because it's starting to exhibit human-like abilities. It's not super surprising that should happen around human levels of compute, and I would personally expect the trend to continue as we scale up towards human level compute and continue improving deep learning efficiency. (I gave this about 50% probability in 2017 before seeing GPT-2, but I've updated significantly in favor over the last 6 years.)

More generally, I think the numbers in your post are wrong and the discussion is somewhat confused. 1e15 to 1e30 is not a narrow interval, I don't think you should compare training costs to inference costs, 1e30 is not the training cost of GPT-3, you should probably compare to brain compute estimates like this one rather than brain emulation estimates...

But I think it's reasonable to step back and say that compared to what you might have expected, biological anchors have been a pretty good guide to ML progress. They are losing usefulness now since at best they have like 10 years of resolution and eyeballing is getting easier and easier as we approach transformative AI. But I still find them helpful as an additional independent check to go along with eyeballing, economic extrapolations, etc. (And until recently I think they were probably the most common way people arrived at in-retrospect-reasonable-looking timeline estimates.)

Both the human brain cost estimates and the GPT3 cost estimates are incredibly noisy/mistaken and I wouldn’t take them too seriously.

To start, 15 orders of magnitude is not a narrow range at all!

For reference, the speed of light is within 8 orders of magnitude of a car, and an Elephant weighs within 6 OOM of a chicken — so this uncertainty is really big.

To be fair, I’ll note that the 10^30 estimate for GPT-3 is clearly an overestimate, the 3 x 10^23 floating point operations is the total compute used to train GPT-3, not it’s per-second usage (the unit is floating point operations, not floating point operations per second. Yes, the notation is confusing.)

I also think that the higher values of 10^27 and 10^30 seem pretty infeasible for brain simulation. But the lower numbers still seems feasible.

Another issue with your estimate is that it’s very bizarre to compare the total cost of training GPT vs the instantaneous operating cost of a human. Surely we want to compare like to like, and compare either instantaneous compute usage, or cumulative lifetime usage (which would multiply the human number by around 9 orders of magnitude).

A final confusion here is how to convert a forward pass to human thinking time. There’s some arguments that a forward pass is way more — you can’t read thousands of characters at once , for example — and some that it’s less — you can generate way more than literally one token at a time, and also do more impressive cognition.

So I think a better estimate for GPT3 cognition in real human equivalent time is something like 10^12 - 10^17 flops, while humans are 10^15 - 10^26 or whatever. This looks way less coincidental!

Most importantly, I think the obvious explanation applies here — to do the impressive kind of human seeming cognition GPT3 and it’s LLM brethren can do, using relatively straightforward methods like neural networks, takes a non-negligible fraction of the human brain’s compute.