Swarm Scaling

Toby_Ord

Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm?

We’ve seen two large and extremely capable swarms from OpenAI in the last few months:

1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced techniques to prevent their actions being logged by OpenAI and 700 of them launched a sophisticated criminal attack on the AI company Hugging Face.
A swarm of 10,000 agents solved a version of the longstanding Navier-Stokes problem in mathematics. It took them just 88 hours to do so, in which time they sent 5 million messages to each other and used 300 billion tokens.

No doubt we will soon see even larger swarms with even more impressive capabilities. But they are not cheap. It is estimated that the swarm of 10,000 agents cost about $20 million at API prices. So while they are very powerful, it will be some time before we see the million-fold reduction in cost needed for this level of power to be possible on a $20/month plan. We should be thinking of it as a grand demonstration of what is possible when money is little constraint — like AlphaGo — rather than a new level of performance for the same cost.

A good way to see AI swarms is as a new form of inference-scaling. The main form of inference-scaling at the moment is having the agent spend more and more time on the task — increasing the maximum length of its chain of thought before it needs to give a final answer. This buys more capability, but at an increasingly expensive price. As measured by performance on maths benchmarks, this performance increases only logarithmically with the amount of compute used. I’ve previously shown that climbing from about 20% to about 80% on a reasoning benchmark typically requires scaling up the length of the chain of thought (and thus the number of tokens, the amount of compute, and the cost) by roughly 100x.

HOW DO SWARMS SCALE?

How do capabilities scale if we instead increase the number of agents in the swarm? There isn’t much data on this — especially for frontier systems like OpenAI’s recent swarms. But OpenAI’s launch post for GPT 5.6 Sol includes some charts containing just enough information to allow one to tease-out an answer.

The chart below shows the performance of three sizes of swarm as their chains of thought are lengthened. Each swarm size displays the usual kind of steep diminishing returns to more reasoning tokens.

Note how the single-agent ‘swarm’ (in light blue) is the most efficient, reaching each level of capability for far fewer total reasoning tokens. Indeed, it appears to use about half as many total tokens as the 4-agent swarm, which uses about half as many as the 16-agent swarm. If we redraw this graph on a logarithmic x-axis, we can see this more easily:

Now we can clearly see that the scaling curve for each swarm-size has logarithmic returns to longer chains of thought (because they are straight lines when plotted on a logarithmic x-axis) and that they have roughly equal slope, meaning that the scaling dynamic remains the same for all these swarm sizes.

We can also see that the 4-agent swarm is stably requiring about twice as many total tokens as the 1-agent swarm to get the same performance, and that the 16-agent swarm is needing roughly twice as many again.

But we don’t yet have a chart that shows how capability increases if we just scale swarm size (leaving the chain of thought length fixed). The experiments OpenAI ran didn’t include this. They didn’t run different swarms of different sizes with exactly the same chain of thought length to see what would happen.

Luckily, we can simulate this from their data. Let’s use the same starting point they did — 1 agent with its lowest reasoning level. Then we ask what would happen if we used a 4-agent swarm with the same average tokens per agent (=4x the total tokens). We can find that point on the 4-agent curve. Because the curve is so straight, the interpolation should be quite reliable. We can then ask what would happen if we scaled up to 16 agents, without increasing the average tokens per agent, by finding the point on that curve with 4x as many total tokens. Let’s plot these in green on the same chart:

We can now see how much we get from purely increasing the number of agents (swarm scaling), and how it compares to purely increasing the length of the chain of thought (duration scaling). The swarm scaling gives a little over half as much gain in capability for the same scale-up of compute. Or put another way, you need to do the scale-up of compute twice to get to the same capability, squaring the total multiplier needed.

Economists have a nice way of thinking about this. They have studied how having many people work on a task can get it done sooner, but usually at the expense of more total person-hours of labour. A convenient way to think about it is that people working together get as much done as one person working for times as long. Here is a parameter measuring how parallelisable the task is. They call it the ‘stepping on toes’ parameter. If = 1, you have a perfectly parallelisable task, with no stepping on toes and no efficiency penalty. But in reality is usually between 0 and 1 — allowing more people to help, but with diminishing returns. For example, if = 0.5 then 100 people working together get as much done as 1 person working for 100 = 10 times as long.

This allows us to state the swarm scaling behaviour more precisely. In the graph above, the slope of the green line is actually 57% the slope of the blue lines, so = 0.57. This means that scaling up the swarm size by 16x would give the same performance as scaling up the length of the chain of thought by just 16 = 4.9x. And if you check the chart, you can see that the light blue single-agent curve reaches the same score as the 16-agent point on the green curve after just a 4.9x scale-up.

The GPT 5.6 launch page includes swarm results for 3 different benchmarks. I asked Claude Opus 5 to determine the values of for each of them. It ran more careful regressions and got values (and confidence intervals) of:

BrowseComp: = 0.68, 90% CI = [0.63, 0.76]
SEC-Bench Pro: = 0.57, 90% CI = [0.52, 0.61]
Terminal-Bench: = 0.48, 90% CI = [0.40, 0.57]

These are very much in line with estimates from economists for the diminishing returns of human teams. The precise value of clearly depends on the kind of task, as we see here with these three benchmarks — some kinds of task are inherently more parallelisable than others. And it may also depend on the scale of the swarm. Here the estimates when scaling up from 1 to 4 agents were similar to scaling up from 4 to 16, but that may no longer be true when scaling from 1,000 to 4,000 — again this will depend on the task. e.g. the task of building 100 brick walls is almost perfectly parallelisable up to = 100, where it becomes much worse.

IMPLICATIONS

Now that we have some preliminary measures of , what do they imply?

First, we can use it to convert between swarm scaling and the more traditional duration scaling. Let’s take the estimates of as 0.48, 0.57, and 0.68. This means that scaling up the number of agents in the swarm by 10x doesn’t get as much performance as using 10x as many tokens with one agent. Instead it gets 10x as much — which is 3x to 5x. And this shortfall accumulates quickly for larger scaleups, with the swarm falling further and further behind. To get the same performance gain as a 100x scale-up of the number of tokens for a single agent you need to scale up the swarm size by 900x to 15,000x.

So why would you ever use swarms?

The most important answer is speed. The 4-agent swarm needed about twice the total number of tokens to get the same performance, but in terms of tokens per agent, it only needed half as many. Since the agents are run in parallel, this means it can theoretically achieve the same task in half the time. And the same was true when moving from 4 agents to 16. In total, one could achieve the task in about 1/4 the time for 4x the cost. In reality, the speedups aren’t quite this good (perhaps because some agents use more tokens than the average), but they are substantial. So for situations in which you’d pay a large premium for speed, swarms can be very useful.

This fits closely with how Noam Brown described it on the Dwarkesh podcast:

Basically if you have four agents working on the problem, it is done twice as fast. So you're basically paying, because there's four agents working for half as long, you're paying a 2x more to get an answer twice as quickly. If you go to 16 agents, you see a similar pattern, it's like a little less efficient, but you continue to see that performance.

More generally, growing the number of agents in a swarm by a factor of could theoretically speed them up by a factor of but uses times as much compute. (Assuming stays constant over that scale-up.)

There may also be other advantages to multiple agents on top of speed. For example, if you keep increasing the length of the chain of thought for a single agent, the performance eventually plateaus. But the height of the plateau for a 1,000-agent swarm may be greater than for a single agent. However, for now, the main demonstrated reason is speed.

A second implication of concerns the possibility of intelligence explosions. I first encountered when studying recursive self-improvement (RSI). In the most common models of RSI, is one of the key parameters for determining whether the rate of growth of AI capabilities explode towards a vertical asymptote. That happens whenever > 1, and is proportional to , so high makes intelligence explosions more likely.

The prominent AI Futures Model for RSI uses a default estimate of = 0.5, while Tom Davidson and Tom Houlden’s median estimate is = 0.6. So these measured values that I’ve derived from OpenAI’s data are pretty much exactly as expected. I’d hoped that the value of for AI agents would be lower, making an intelligence explosion less likely, but that appears to not be the case. People should keep tracking this as new estimates for appear and (especially) when new orchestration methods increase the value of for a given type of task.

THE NAVIER-STOKES SWARM

When OpenAI announced their 10,000-agent swarm had solved the Navier-Stokes problem, they included this graph:

View fullsize

They have done their best to remove most of the useful information from this graph (such as the x-axis labels) and they don’t even say what form of inference compute is being scaled. But given that they went all the way up to 10,000-agent swarms, I’d bet it is tracking the number of agents in the swarm (i.e. that it is tracking total tokens spent, but that the main difference between data points is the number of agents in the swarm rather than the tokens per agent).

One thing I immediately noticed was that while it has the usual logarithmic performance gains, the slope of the curves is about half that of the inference-scaling curves for o1 and o3. So instead of requiring a ~100x scale-up of compute to go from 20% to 80% on the benchmark, it is requiring 10,000x the compute. I first wondered if this was due to the unusual benchmark of open math problems — perhaps the standard deviation of their difficulties is twice that of the problems in the AIME maths benchmark or the ARC-AGI-1 benchmark. That could still be right, but since we’ve now seen that is around 0.5, this alone would be enough to perfectly explain the halved slope of these scaling curves.

The other interesting thing about this chart is that the jump up from GPT-6 Astra’s performance to the performance curve of the internal model is much larger than we’ve seen before. The jumps from o1 to o3 and from o3 to GPT-5 were enough to allow the better model to get the same performance as the prior model using about 1/3 as many tokens:

View fullsize

But on their new chart, the internal model is getting the same performance as GPT-6 Astra for about 1/100 the tokens. That’s like 4 previous jumps in one. Even if we adjust for the slopes being lower due to swarm scaling, it would still be 2 jumps in one. Whatever changed was a big deal.

Indeed on the Dwarkesh podcast, OpenAI’s Noam Brown took pains to explain that the dramatic success of solving a Millennium Prize problem wasn’t primarily due to the large scale of the swarm, even though that was the part that seemed most unusual with their setup:

I wouldn't even attribute 10% of the credits to multi-agent. The reality is OpenAI has trained a very powerful model.

From the data they’ve released, I think that’s right. The multi-agent swarms helped them go fast enough to scoop Anthropic (and academia) by getting the result in just 88 hours, but it probably made the project much more expensive too. e.g. if we (somewhat heroically) assume = 0.5 at all points in the scaleup from 1 to 10,000 agents, then they could have got the same result for 10% of the cost in 10x the time (37 days) by using 100 agents, or at 1% of the cost in 100x the time (1 year) using 1 agent.

There's reason to expect λ can be substantially increased, and maybe even brought above 1,^[1] by the right orchestration and for some tasks, even keeping in mind the good caveats you give about ex. the Navier-Stokes swarm.

"Scaling Discovery through Test-Time Communication" announced today finds that while swarms aren't too useful on ex. Terminal-Bench, even Sonnet 4.6 swarms can outperform independent agents with equivalent compute budget, specifically when the task is verifiable and requires some creativity (ex. ARC-AGI-3, FrontierCS Polyomino Packing, MNIST Compression).

Further caveats:
- They only tested up to 5 agents, so not exactly an OpenAI-style 10k swarm, and it's likely there are diminishing returns past some N of agents.
- The ARC-AGI-3 result is less exciting than it looks because it's counting that benchmark's "actions", not the tokens to perform them.
  - Otherwise Fig. 4 would actually demonstrate a case where λ > 1!
- Below a certain minimum compute budget (necessary for communication overhead?) swarms do worse.
- The paper's own model caps λ at 1.
- It's possible that swarm advantages decrease with stronger models, though per Kim et al. this might be because stronger models can just solve the tasks outright and thus benefits less from sharing intermediate results. Today’s paper by Park et al. specifically chose to pick a weaker model to run ARC-AGI-3 on.
  - In other words, swarms aren't useful if a single agent can just one-shot the task. Sort of how like you're better off putting one programmer on a task they can complete in an afternoon, than putting 5 on it.
Un-caveats:
- Some tasks could only be completed by communicating agents.
- Agents were all the same in each test, so very homogeneous, but still managed to make and share useful individual breakthroughs via a bit of clever protocol work.

Re: λ = 0.5 (like AI Futures Model) being "very much in line with estimates from economists for the diminishing returns of human teams", I think this is the wrong anchor.

On priors, human teams can have λ > 1 in some cases; that is, there are tasks where having four people work the problem for a week is better than having one person work the problem for a month.^[2] The advantage comes from pursuing different directions and sharing intermediate results/creative ideas, based on individual experience/skills. I think this sort of thing is why Anthropic is now assigning persistent identities to their 30,000 internal agents.

I suspect the main thing dragging λ down historically is agent homogeneity, but agents also have potential advantages in coordination that humans don't, like better trust mechanisms, lower transaction costs, skill/memory sharing, and deliberate usage of decision theory.

Finally, to do handwavy analysis of your backed-out λ values:

BrowseComp:   = 0.68, 90% CI = [0.63, 0.76]
SEC-Bench Pro:   = 0.57, 90% CI = [0.52, 0.61]
Terminal-Bench:   = 0.48, 90% CI = [0.40, 0.57]

It's notable that Terminal-Bench showed no swarm benefit in the paper above, and that's also the lowest λ you found. The other OpenAI 16-agent cases were also using Ultra mode, which by my understanding is more of a traditional hierarchical multi-agent system than the freely-communicating systems that Anthropic and OpenAI are moving towards. So we might expect this to translate into:

Tasks not suited for swarms (ex. very serial): λ ≈ 0.5
Tasks where hierarchical multi-agent swarms are useful: λ ≈ 0.6 or 0.7
Tasks where communicating swarms can share meaningful intermediate progress: λ ≈ 1
After some future advancements in agent coordination (where task is not trivial for single agent): λ > 1

^{^}
Meaning superlinear returns to adding more agents, given fixed test-time compute. That is, rather than more parallel agents giving you diminishing returns to capabilities, more parallel agents give you increasing returns to capabilities, at least within some useful bounds.
^{^}
Or equivalently, four people spend 10 hours on the problem within a week vs. one person spends 40 hours within a week.

Agents are only now starting to be trained to cooperate with each other and self-organize.

In my experience today's agents frequently make obvious coordination mistakes and fail to use good theory of mind or have high-trust patterns of communication when prompting each other. They feel like they are optimized for doing everything themselves and at best delegating very specific tasks to subagents.

I haven't looked into SEC-Bench Pro, but if you take a benchmark that can be done pretty well by a single agent, it makes sense that it's not that parallelizable, but things like "running a large corporation" clearly can't be done without some mechanism to divide attention. For LLMs, things work somewhat differently, since you can clone agents and share context more cheaply, but it is definitely not going to make sense to chuck all the inputs of your company into a single smart agent's context and have them make every decision.

Also, time does matter. If an agent could have solved Navier Stokes by carefully maintaining a large file system for itself and working on each subproblem one at a time, maybe it could produce the whole Lean proof in ~5 years. There are many kinds of problems where that would not be a success.

Civilization basically just is effective mechanisms of dividing labor so that a bunch of people can focus their finite attention/context windows on doing one thing well at a time. It would not work if you only had one guy trying to do everything at once, even if that guy were really competent.

I think humans kind of suck at cooperating with each other and self-governing, compared to what is possible. The difference between the least effective teams of humans and the most effective teams, comprised of all the same people with different communication norms, already spans orders of magnitude, but even the most effective humans are more limited in trust and communication bandwidth than LLMs could/likely soon will be.

Thanks Julian, that's very thoughtful. I've now read your post on this and I thought it was good.

I'm definitely less bullish than you are about getting λ>1, though it may be possible, especially in cases where agent diversity is important (and where that can be implemented somehow) or where it is hard for a single agent to try out lots of different approaches successively without them contaminating each other.

You may be interested to know that Phil Trammell is thinking about something similar to increasing λ in the essay below. In fact he sees understanding parallelisability as key to understanding the prospects on an intelligence explosion:

https://philiptrammell.com/static/Parallelizability.pdf

Ah, a much more formal take that decomposes the relevant factors, I gather. I'll look into that, thanks!

Edit: also FYI I haven't looked deeply yet, but Opus 5.5 System Card would seem to still show λ in the 0.5 to 0.7 range, like I imagine you would predict.

Maybe a naive question, I'm having trouble understanding how λ > 1 could be possible in principle; if a 10 agent swarm can do 11 times the work, then could I not improve my single agents by serializing this parallelization? (I think I'm just parroting some bog-standard argument in parallel computing.) That is, have my single meta-agent act as agent 1 for a while, then agent 2 for a while, and so on, the same way a single-core CPU simulates multi-threading.

Initially I thought that a swarm might win out against the above setup in efficiency by saving on the cost of context switching, but I would guess that loading up a new agent has the at least the same (and possibly higher) cost as context switching an existing one?

No that's a great question, thanks! The paper I discussed above actually agrees with you, that's why their model of how this works caps λ at 1.

My counterargument is that this is true for an idealized serial agent. We can refer to the human example again for intuition:^[1]

There are tasks where having 4 people work the problem for 10 hours each in a week is better than having 1 person work the problem for 40 hours in that week.

We can see that λ > 1 could be true in a couple core scenarios:

The 4 people have different skillsets that complement each other, while the 1 person has just one skillset.
All 5 people have the same skillset, but they take different approaches due to random chance/coordination/initial conditions/experience, and one of those approaches is more fruitful. Assuming a large number of possible approaches, it's more likely the team finds a better one.
1. With optimal approach switching, the 1 person could attempt four different approaches in the same amount of time, making it even odds, but that isn't really how people work in practice. Idealized vs. real.

You can also see here why agent homogeneity is so problematic.

Re: efficiency on context switching, I think you're right that swarm has no efficiency there, optimally. But of course one of agents' main flaws (and humans' main flaws?) is getting too attached to the current context/approach/belief.^[2]

^{^}
Reformulated to use version from footnote of previous comment, which in retrospect I prefer to avoid confusion about task speedup via parallelization, which is a separate matter.
^{^}
Though you could just employ a setup that wipes the serial agent's context entirely, and denies it access to previous work. Luckily we can't do that to humans!

I see, I agree that it's possible that a particular AI design easily gets caught in its context and periodically wiping it like an anterograde amnesiac might actually make it do better.

I guess it feels like if we ever had a model that was λ > 1 that would signal some serious inadequacy of the model that feels not hard to RL out?

A good theoretical question! If you have a set of humans with λ > 1, is that a sign of serious inadequacies in some or all of them, which ought to be addressed? Possibly!... if you want to optimize for task success, anyway.^[1]

Thinking more about our context-switching serial agent:

On tasks where sharing intermediate results is very valuable (ex. hard research tasks), you really want the "trained to message other agents about intermediate results, and know how to trust/accept/reject those messages" behavior Anthropic/OpenAI are currently pursuing. Theoretically you could implement that by just having the serial agent send messages to its future selves. But the RL to teach that is multi-agent system RL! And the easiest implementation is real-life swarms.^[2]

To put that another way, I expect we'll have agents trained to send messages in a multi-agent system in an effective way. If you run a practical test with just one of those agents that has no one to message, against the multi-agent system, then you can get λ > 1. There will be a theoretical setup in which a serial agent matches the multi-agent system performance, but it won't be used in practice.^[3]

Thinking more about heterogeneity:

There's an interesting path where λ > 1 in variants of a single model. A possible future: we have some basic continual learning using overnight weights-updating for agents working different jobs/roles, a more advanced version of Anthropic's current long-running agent identities setup. There is no single agent that knows everything and that could substitute for all of them, because getting all of that context into the weights crowds model identity/skills/context/J-space in some way that makes them less effective.^[4]

That all said, you've clarified my thinking here and I've updated down on λ > 1 thanks to this conversation.

^{^}
In the limit, this almost feels like talking about whether a single grand unified consciousness is superior to a civilization, or an ASI singleton vs. ASI swarms, though we would have to carefully decompose "superior" there.
^{^}
Though you could implement this in your serial agent by giving it the ability to end its current session with a "wait for reply/new message" tool. There might be some tricky caching issues with that but perhaps surmountable.
^{^}
What does this mean for forecasting? I think the practical effects here mean that the practical speedup can still be equivalent to λ > 1. That is, parallelism can deliver increasing marginal returns (again, at least within some bounds on N agents) on forecasted available compute compared to the old pre-swarm baseline. However, you could just adjust the baseline to the ideal multi-agent-system-simulating serial agent, which clamps λ ≤ 1.
^{^}
But it's plausible no such limit exists in practice, or that more heterogeneity than this is needed, aka you need substantially different training runs to really get near-human variety.

Anything constrained by total working memory is a candidate

I think an ideal serialized agent wouldn't be more constrained by working memory than a swarm would be. It's not handling all the context all at once, it's context-switching. It can emulate the full swarm (and even messaging behavior!) that way, see more discussion of that here.

But then again, we saw that kind of logic break down in practice in multi-threading on CPUs because of caching. In theory, a single-core CPU can emulate an N-core CPU in N× as much time. In practice, constantly swapping out the whole CPU L1/L2 cache wastes time, so a single-core CPU takes longer. Or rather, an N-core CPU is more than N× better; λ > 1.

So this comes back to the discussion on context-switching costs. The simple case is that you switch once per swarm agent the serial agent is simulating, and in that case caching costs can be equal. But the organized swarm case requires switching most every time the serialized agent wants to send or receive a message, which would be a lot more cache-swapping than running the agents in parallel... and with a lot bigger caches than CPUs have. So ex. if you have a nearly-full context window of 900k tokens, and you send a message after every 5k tokens of reasoning before swapping out the whole cache, you're in trouble!^[1]

However even then, I suspect this might be less relevant than heterogeneity.

^{^}
I don't think clever KV-cache swapping is gonna save you here either when you're trying to simulate 10,000 agents. Sorry Deepseek!

Not-too-important Addendum: another paper I saw today: "Self-Organizing Agent Teams Learn to Reason Together". The headline relevant result here is:

Self-Organizing Agent Teams (SAT) outperform their strongest member, compute-matched inference by the strongest individual agent, and perfect routing over independent answers.

This sounds like λ > 1, but I think it's more an exercise in caution on skimming arXiv preprints, as I don't think this is good evidence:

Their linearization test is executing the strategies developed by the multi-agent, multi-model team of 3. Not apples to apples. A multi-agent, single-model team of 3 does worse* (though frustratingly it's also executing given strategies). Performance gains may partly come from heterogeneity.^[1]
- Though Table 4 tries to demonstrate they come at least partly from the self-organized team structures.
- *Except for one benchmark (BIG-Bench Extra Hard) where it does better for some reason.
- Also "does better/worse" is quite possibly mostly inside confidence intervals here... which aren't reported. (!)
"swarm" sizes here are just 3, and models are old (ex. Sonnet 4). Sample size also kind of small.
Compute-matching is loose; no token counts given, word "approximately" is used.
Headline result applies only to some of the benchmarks they tested.

On the whole I'm not comfortable updating much on this evidence. (also in terms of self-organizing teams results, unfortunately these folks were scooped by a more interesting and relevant result from Anthropic's Opus 5.5 System Card, Ctrl-F for "Emergent team structures")

^{^}
You know I've found it useful myself to compare/contrast multiple LLMs' results on programming tasks. I wonder if there might be gains to fine-tuning a lot of variants of a single frontier model, or if you really need different pretrains to get value.

They have done their best to remove most of the useful information from this graph (such as the x-axis labels) and they don’t even say what form of inference compute is being scaled. But given that they went all the way up to 10,000-agent swarms, I’d bet it is tracking the number of agents in the swarm (i.e. that it is tracking total tokens spent, but that the main difference between data points is the number of agents in the swarm rather than the tokens per agent).

FYI, my friend at OpenAI says this interpretation is incorrect and we are just comparing single agents given more compute.

Thanks — that's very interesting. If I hear that from another source I'll remove or modify that section.

I expect multi-agent scaling to be about as big a new scaling law, and unlock as many qualitatively new capabilities, as the reasoning paradigm. It seems like it's just starting to really work, but there's tons of headroom.

Biology and human civilization have lots of examples of what becomes possible when organisms, even relatively unsophisticated ones, can productively coordinate in large groups.

Even if we do pause development of smarter, larger models and focus on alignment, progress on multi-agent coordination will likely continue, if only as a byproduct of alignment, so we are still going to see major qualitative capabilities gains.

I think it will be important, especially once extending CoT has gone so far that you get many compactions and the additional diminishing returns that entails. Also important when you need speed and are willing to have a less efficient system to get it. I think it will be big, though not as big a thing as reasoning. But I think it will be big enough (and shiny enough) that many people will think it is as big a thing once it is here.

Good post.

It's possible for the gains to compute efficiency at reaching some given level of performance from multi-agent scaling to be large, even in worlds where λ is < 1, if it's possible to anticipate which particular agents in the flock require more compute and distribute compute accordingly.

Analogously: If you're optimizing for capital-efficiency while trying to accomplish some task in a human organization, you don't just hire people like Jeff Dean, Ilya Sutskever, and so on, even if people at their level individually might be the best for every role in an organization. Instead you hire a few people at their level for a few roles in the org chart, and then some (much less expensive) people everywhere else.

Similarly, if you're optimizing for compute-efficiency while trying to accomplish some task with an AI agent flock, you could distribute compute between the largest and most massively expensive AI out there and some less smart, but much less expensive AIs. And it seems likely that, as differential hiring increases capital efficiency, this would increase compute-efficiency.

I agree that things like this should effectively lower the costs of swarms. However, I note that they are also true for single-agent runs. e.g. for some tasks you should use a cheaper model, or use a cheaper model to do one part and then a smart model to analyse the results. So while it is important and under-applied in current AI systems, it applies to both the single agent and multiagent costs that are used to estimate λ, and not clear it changes λ itself. I suspect it does raise it a bit overall.

Speed is quite valuable - e.g. for AI R&D automation, and for getting new results/evals out of new models faster.

I agree that speed can be quite valuable.

These are very much in line with estimates from economists for the diminishing returns of human teams.

Which estimates are you referring to?

These are the kinds of numbers I saw back when reading the semi-endogenous growth theory literature and the RSI models inspired by it. But there seems to be surprisingly little evidence pinning it down. Apparently some methods only pin down the ratio r (= λ/β), not λ on its own. But I'd have thought old econ work on production lines and other forms of labour organisation should have lots of data on this. e.g. just get ten large tasks and try them with groups of 1, 2, 4, 8, 16, 32, 64 people and see how long they take. And do similar things with tasks that have already had years of optimisation pressure to find the efficient ways to divide them up.

Yeah I had a hard time finding good numbers too, even in software engineering. But only spent ~15min checking.

Curated. There's a kind of OP (overpowered) skill for prediction that is extrapolate from the data seen so far, and this feels like a great example of doing that. Even as swarms are only becoming a "thing", I appreciate this post for thinking about where that's going. The idea of a "new kind of scaling" is a not something that had clicked for me previously.

I wonder if there were be another kind after this, or this perhaps the final level. (More than three degrees of recursion??)

I personally had a small project (an emergency site database migration under heavy traffic) where I used a N=4 swarm of Claudes to get it done in a hurry. Watching them work cooperatively was remarkable. I don't know if there was secretly some harness, but the models took to it very readily. I could have believed they'd been trained for it. Following that, incidents like HF had a visceral referent for me, and I'd encourage others to set up circumstances to observe it too.

All in all, good post. Kudos.

I found the suggested Sqrt(N) speed up in swarms of size N (and human teams of size N) kinda intriguing, as it reminded me of the Sqrt(N) speed up in Grover’s search algorithm when creating a quantum superposition of N instances.

Given the discussion about potential lambda values significantly higher than 1/2 elsewhere in the thread, I’m not reading much into this. Just a Hmmmm.

Further caveats:
- They only tested up to 5 agents, so not exactly an OpenAI-style 10k swarm, and it's likely there are diminishing returns past some N of agents.
- The ARC-AGI-3 result is less exciting than it looks because it's counting that benchmark's "actions", not the tokens to perform them.
  - Otherwise Fig. 4 would actually demonstrate a case where λ > 1!
- Below a certain minimum compute budget (necessary for communication overhead?) swarms do worse.
- The paper's own model caps λ at 1.
- It's possible that swarm advantages decrease with stronger models, though per Kim et al. this might be because stronger models can just solve the tasks outright and thus benefits less from sharing intermediate results. Today’s paper by Park et al. specifically chose to pick a weaker model to run ARC-AGI-3 on.
  - In other words, swarms aren't useful if a single agent can just one-shot the task. Sort of how like you're better off putting one programmer on a task they can complete in an afternoon, than putting 5 on it.
Un-caveats:
- Some tasks could only be completed by communicating agents.
- Agents were all the same in each test, so very homogeneous, but still managed to make and share useful individual breakthroughs via a bit of clever protocol work.

Re: λ = 0.5 (like AI Futures Model) being "very much in line with estimates from economists for the diminishing returns of human teams", I think this is the wrong anchor.

Finally, to do handwavy analysis of your backed-out λ values:

BrowseComp:   = 0.68, 90% CI = [0.63, 0.76]
SEC-Bench Pro:   = 0.57, 90% CI = [0.52, 0.61]
Terminal-Bench:   = 0.48, 90% CI = [0.40, 0.57]

Tasks not suited for swarms (ex. very serial): λ ≈ 0.5
Tasks where hierarchical multi-agent swarms are useful: λ ≈ 0.6 or 0.7
Tasks where communicating swarms can share meaningful intermediate progress: λ ≈ 1
After some future advancements in agent coordination (where task is not trivial for single agent): λ > 1

^{^}
Meaning superlinear returns to adding more agents, given fixed test-time compute. That is, rather than more parallel agents giving you diminishing returns to capabilities, more parallel agents give you increasing returns to capabilities, at least within some useful bounds.
^{^}
Or equivalently, four people spend 10 hours on the problem within a week vs. one person spends 40 hours within a week.

Thanks Julian, that's very thoughtful. I've now read your post on this and I thought it was good.

https://philiptrammell.com/static/Parallelizability.pdf

No that's a great question, thanks! The paper I discussed above actually agrees with you, that's why their model of how this works caps λ at 1.

My counterargument is that this is true for an idealized serial agent. We can refer to the human example again for intuition:^[1]

There are tasks where having 4 people work the problem for 10 hours each in a week is better than having 1 person work the problem for 40 hours in that week.

We can see that λ > 1 could be true in a couple core scenarios:

The 4 people have different skillsets that complement each other, while the 1 person has just one skillset.
All 5 people have the same skillset, but they take different approaches due to random chance/coordination/initial conditions/experience, and one of those approaches is more fruitful. Assuming a large number of possible approaches, it's more likely the team finds a better one.
1. With optimal approach switching, the 1 person could attempt four different approaches in the same amount of time, making it even odds, but that isn't really how people work in practice. Idealized vs. real.

^{^}
Reformulated to use version from footnote of previous comment, which in retrospect I prefer to avoid confusion about task speedup via parallelization, which is a separate matter.
^{^}
Though you could just employ a setup that wipes the serial agent's context entirely, and denies it access to previous work. Luckily we can't do that to humans!

Thinking more about heterogeneity:

That all said, you've clarified my thinking here and I've updated down on λ > 1 thanks to this conversation.

^{^}
In the limit, this almost feels like talking about whether a single grand unified consciousness is superior to a civilization, or an ASI singleton vs. ASI swarms, though we would have to carefully decompose "superior" there.
^{^}
Though you could implement this in your serial agent by giving it the ability to end its current session with a "wait for reply/new message" tool. There might be some tricky caching issues with that but perhaps surmountable.
^{^}
What does this mean for forecasting? I think the practical effects here mean that the practical speedup can still be equivalent to λ > 1. That is, parallelism can deliver increasing marginal returns (again, at least within some bounds on N agents) on forecasted available compute compared to the old pre-swarm baseline. However, you could just adjust the baseline to the ideal multi-agent-system-simulating serial agent, which clamps λ ≤ 1.
^{^}
But it's plausible no such limit exists in practice, or that more heterogeneity than this is needed, aka you need substantially different training runs to really get near-human variety.

Anything constrained by total working memory is a candidate

^{^}
I don't think clever KV-cache swapping is gonna save you here either when you're trying to simulate 10,000 agents. Sorry Deepseek!

Not-too-important Addendum: another paper I saw today: "Self-Organizing Agent Teams Learn to Reason Together". The headline relevant result here is:

Self-Organizing Agent Teams (SAT) outperform their strongest member, compute-matched inference by the strongest individual agent, and perfect routing over independent answers.

This sounds like λ > 1, but I think it's more an exercise in caution on skimming arXiv preprints, as I don't think this is good evidence:

Their linearization test is executing the strategies developed by the multi-agent, multi-model team of 3. Not apples to apples. A multi-agent, single-model team of 3 does worse* (though frustratingly it's also executing given strategies). Performance gains may partly come from heterogeneity.^[1]
- Though Table 4 tries to demonstrate they come at least partly from the self-organized team structures.
- *Except for one benchmark (BIG-Bench Extra Hard) where it does better for some reason.
- Also "does better/worse" is quite possibly mostly inside confidence intervals here... which aren't reported. (!)
"swarm" sizes here are just 3, and models are old (ex. Sonnet 4). Sample size also kind of small.
Compute-matching is loose; no token counts given, word "approximately" is used.
Headline result applies only to some of the benchmarks they tested.

^{^}
You know I've found it useful myself to compare/contrast multiple LLMs' results on programming tasks. I wonder if there might be gains to fine-tuning a lot of variants of a single frontier model, or if you really need different pretrains to get value.

They have done their best to remove most of the useful information from this graph (such as the x-axis labels) and they don’t even say what form of inference compute is being scaled. But given that they went all the way up to 10,000-agent swarms, I’d bet it is tracking the number of agents in the swarm (i.e. that it is tracking total tokens spent, but that the main difference between data points is the number of agents in the swarm rather than the tokens per agent).

FYI, my friend at OpenAI says this interpretation is incorrect and we are just comparing single agents given more compute.

Thanks — that's very interesting. If I hear that from another source I'll remove or modify that section.

Good post.

Speed is quite valuable - e.g. for AI R&D automation, and for getting new results/evals out of new models faster.

I agree that speed can be quite valuable.

These are very much in line with estimates from economists for the diminishing returns of human teams.

Which estimates are you referring to?

Yeah I had a hard time finding good numbers too, even in software engineering. But only spent ~15min checking.

Given the discussion about potential lambda values significantly higher than 1/2 elsewhere in the thread, I’m not reading much into this. Just a Hmmmm.

208

Swarm Scaling

208

HOW DO SWARMS SCALE?

IMPLICATIONS

THE NAVIER-STOKES SWARM

208

208