For starters, this is an awesome post and thank you for writing it.
My preferred approach for dealing with these apparent contradictions:
The exact right balance may vary by individual.
Examples at MATS:
Unfortunately I suspect that recent evidence indicates that we should be leaning into more of a war frame.
Of course, there's a separate question of alignment difficultly. Honestly, I'm more worried about the possibility that we have an insufficient number of people trying to dramatically slow things down or trying to develop more principled solutions than I am that we aren't taking a broader, civilisational approach.
I've thought about this a lot. The best answer I can come to is:
We have enough people such that crunch time efforts are relatively saturated
I would be surprised if this were the case. Maybe we're bottlenecked on co-ordination. If so, I think we should try to rapidly develop new co-ordination mechanisms.
Even if crunch time efforts aren't saturated, the right allocation of people is to put some people on efforts that only work if crunch time works... But we have no new ideas or leads, and have to crunch time harder, until we die.
How much will "civilisational" efforts help with that?
[My own opinions below - I talk about MATS here but this should not be construed as a MATS position]
There are useful frames here, but I think some of the confusions/tensions you highlight above are somewhat resolved by a sensitivity to the particular threats, risks, and disruptions that we aim to address in different contexts.
Most of us expect a cluster of related near-term risks: loss of control, extreme concentration of power (in the labs or in autocratic power grabs), cyber and bio misuse threats, international crises resulting in nuclear exchanges etc. The "war frame" you describe is probably more correct for threat models of this kind. I wish it were otherwise, but it would be a a loss for all of us if e.g. the people best-positioned to help effect a pause or slowdown right now largely elected to avoid sprinting. It's a painful and heavy burden, and we should extend grace and understanding to anybody who isn't in a place to do this kind of work. But it's not unreasonable for somebody to assess the threat model of their problem space, conclude that their work has a deadline, and try to push hard to mitigate the acute risks.
But while the near-term tasks are pretty pressing, I agree that we probably need vastly more people still to help us navigate a chaotic and uncertain future full of nasty pitfalls. Whatever equilibrium our world has had has popped like a balloon, and I don't expect any point of stability again in my lifetime. But even narrowing our focus to omit all the institutions we will need to reinvent and all the disruptions we will need to resolve, there are a great many agendas within direct "mitigating the catastrophic risks of AI" work that require deep, creative thinking, longer time horizons, expansive freedom, opportunities for diverse collaboration, and convenings to argue through branches of the argument tree that we've newly opened.
We need a mix of both, but this phrasing obscures more than it helps. More precisely, we have a mix of problem spaces to address, and need to match the nature of the work to the nature of the problems. In problem spaces where uncertainty is very high, I'd advocate for a mix of approaches and timelines as a consequence of that local uncertainty.
The pipeline considerations for all this are interesting. We know that there is a tail of impact: a few people or orgs end up having an outsized influence on future outcomes. But we almost never know who they will be in advance. And (as Richard Ngo's recent posts have highlighted) influential agendas can sometimes have the opposite effect as the one that was originally hoped-for. Given the uncertainty here, the only way I know for sure that we lose is if we make a single undiversified bet (on one mode of work or one candidate shape etc).
Fellowships are competitive, but for the most part we're all scaling very hard to accommodate all the great people applying. MATS has moved to three cohorts a year instead of two, and is currently in the middle of our first Autumn cohort. And at the same time, MATS is aiming high with increases in individual cohort size and in the ambition of the support we provide per fellow (e.g. recently doubling our compute support). And we launched the MATS residency explicitly to support both rare/exceptional expertise transitioning into AI Safety, and to provide a research institute context for long time-horizon high-freedom work (alignment moonshots etc).
Yes, individual orgs are often very selective. In part, it seems like finding great generalists and managers and experienced strategists to help them scale effectively is part of the problem. But, like, I would be shocked if we are anywhere close to saturating the marginal benefit of additional capacity working in AI Safety. Like, I doubt we're within several orders of magnitude. If anything, I feel like we as a field are constantly dropping balls because there aren't enough hands. It's like a daily occurrence that nobody has a good answer to questions that would be helpful to know, or that I encounter new project or experiment ideas that almost certainly nobody is going to work on any time soon because of capacity constraints. Standing up new orgs to catch some of this work seems good, but most of the startups I know are desperate to hire and can't find enough interest. The fact that our ecosystem has responded to this state of affairs by focusing on building capacity, and that the rest of the world has responded to this by increasingly trying to bring their talent into AI Safety, largely all seems like the correct dynamic.
The “War mode vs Civilisation mode” dichotomy definitely resonates. Do you think the field should develop pipelines for both?
Right now, pivoting into AI safety from another career feels like being pushed into war mode by default: everything is urgent, we need several publications and artefacts and we’re already running behind. Like you mention in your footnote 2, career accelerators are getting more and more out of reach as recommended entry-points favour people who are already ready to contribute and push others to produce more artefacts/research to stay competitive.
Without wider entry-points, the avalanche of research directions and feeling of impending doom leaves little room to think, ponder and connect needs of the field with ideas and results.
I’ve personally decided to stop letting the timelines dictate my learning pace. To stop falling into the “produce more to get a slot“ spiral I’m taking my time with the fundamentals, drawing connections between fields, and updating my view of the long-term even with uncertainty looming.
Yes, although I feel even the word "pipeline" puts us in a frame of guided development. But civilisations must allow us to meander, dream loftily, contemplate blue skies. A great blogpost can do a lot more than 10 NeurIPS papers, even if the second is more likely to get someone accepted to a fellowship. I think something good might be a system or a playground of sorts that allows people to sort into what their temperament suits; some prefer engineering and others slow daydreaming. Perhaps one of the fellowships is suited for this, although I haven't looked deeply enough into each of them to decide.
From my limited experience with applications, some programs like LASR, ERA and Generator Residency pair mentees and mentors up after admission, and after a couple weeks of exploration and discussions. That could be a good thing to generalise to sort people according to temperament, instead of applying to a specific mentor like in SPAR, MATS or Pivotal, and most others.
Image by Claude Opus 5.5 using Python (NumPy and Pillow)
This is written in my free time and personal capacity.
Two Frames
In AI safety, there are a couple of tensions at the heart of the field, which are interrelated.[1]
Talent: we are talent-constrained and must recruit. But jobs and fellowships turn away nearly everyone.[2]
Field size: we must expand the field and bring more people in. But the field is flooded with slop papers in conferences. And, most papers have a small audience, and aim to shift the practices of roughly five labs (and usually some subteam within them).[3]
Status: we must avoid seeking status, as it disorients, corrupts. But, we must make the field legible and attractive to outsiders.
Raising the bots: we must deceive the AIs to control them and evaluate them without confounds, obscuring what they know about their own situation. But, we hope to gain their trust, must rely on them to do good work at pace, shape their character and retain good relations.
Speed: we must research in a slow mode, think deeply about the most important problem, do things to aim at high assurance. But it is crunch time. It was crunch time two months ago. We must make hacky progress now and trust that scale will clarify things later.
I think these tensions are, at least in part, rooted in what I will call a "war" frame and a "civilisation" frame. These frames are oriented around a different end goal, or telos of AI safety.
In the war frame, AI safety is a campaign with a deadline, with decisive action that needs to be taken at a handful of labs over the next few training runs. In this frame, your job squarely focuses on changing what happens in those labs before then.
In the civilisation frame, AI safety is closer to dealing with a new population of aliens arriving among us. If aliens landed, we wouldn't ask how many alien researchers we need; we would need a counterpart to nearly every human institution: diplomats, courts, customs, doctors, police. By that measure, the right number of people working on AI is huge, and most of them would not be researchers.
This is more than just timelines, of course, and I find it helpful to think of it as a broad, sociological orientation. One observation is that in terms of talent, we tend to recruit under a civilisation frame, but allocate resources and slots under a war frame. In raising the bots, the war frame treats AIs as systems to test and control, so deceiving them is one tool among many. But the civilisation frame treats them as counterparties we will live alongside. You can't run sting operations indefinitely if you want to build trust!
On the War Frame
I think the war frame is prevalent. You can hear it in a lot of talk: crunch time, timelines, the next training run. Often, I catch myself using this language! Boyd Kane accurately describes many serious people as being in "war mode": endlessly driven, not by money or promotions, with everything else losing its interest (MATS 9 Retrospective & Advice). Part of the reason it prevails, I think, is that AI safety is organised around a failure it wishes to end.
But that alone needn't make a field transient. Medicine, public health and computer security all exist to remove something. And these are lasting fields since the problem is chronic. Security is hard to solve, and security needs grow with every new deployment. Large things that did dissolve on success were one-shot projects, like smallpox eradication. In many ways, AI safety has the trappings of a field (fellowships, careers, conferences) but seriously imagines itself as a project with an end date, like a global Manhattan Project.
Near Horizons
I now discuss some personal unhappiness I have with the war frame. Chess engines have a failure called the horizon effect. An engine that can only search a few moves ahead will push an unavoidable loss just past what it can see, so the loss seems to vanish. War mode does the same with everything after the crunch: abrading away your career, your health, your purpose.
One observation is that war mode can narrow your horizons in a corrosive way. When you start in AI safety and you go into war mode, it's hard to reflect; you're hurtling. In the comments on that MATS retrospective, Kyle O'Brien says every day off with his family now feels like a trade-off he may come to regret (MATS 9 Retrospective & Advice).
If you work assuming you are ephemeral, you may sputter out even if we do get a pause! A pause turns a sprint into a marathon, and war mode leaves you out of fuel to run a marathon. One post on psychological support for safety researchers goes as far as suggesting it may be rational to plan to burn out just as ASI arrives (Psychological support for AI safety researchers is neglected).
And: working on the assumption that you will obliterate yourself can't be healthy, even if we do (IMO, implicitly) think highly of ourselves for knowing about the problem and doing something about it. Perhaps the status is part of what keeps it bearable: being important now stands in for having a future. I recall a striking post by Josh Clymer, opening by saying their goal as a safety researcher is to work themselves out of a job, and continuing that safely handing the work to AI should be the field's primary end goal (How might we safely pass the buck to AI?). The AI should be safe, and if it is safe, it can do our job.
That's a rough spot to be in! And I reject it, repudiate such hopelessness: even if we are destined to fail, we can dream of frames that aren't hostile to our wellbeing.
The Civilisation Frame
Sometimes, on long car rides, I think about two decades from now. What does a world look like where AI safety is a chronic rather than a monumentally ephemeral problem, and we treat it more like security or public health? As Claude often says: a thing can also be another thing. A role transitional without the life provisional; scaffolding comes down when the building is built, but the builders go on, building other offices, churches, homes.
Pieces of this frame already exist in some form, mostly as proposals for how to relate to AIs. One way to think of AI risk is as an intensified version of the old problem of balancing power among agents with different values, where our political traditions have a lot to offer (Otherness and control in the age of AGI). One could give AIs the contract, property and tort rights that corporations already have, on the logic that trade beats war (AI Rights for Human Safety). Alignment can be framed as driver training, which still leaves us needing the traffic lights (Infrastructure for AI Agents).
What does this frame imply for our orientation as a field?
For starters, alignment will likely only be one of many Herculean challenges we'll have to overcome. Humans could lose influence over the economy, culture and the state even if efforts to stop AIs from scheming or seeking power succeed (Gradual Disempowerment). In the war frame, success means that the troops can return home, a field to stand down, return to reserves. In the civilisation frame there is no stand-down; we must build peace, society, answering thorny questions of how humans and AIs share institutions over decades.
While we must prepare to fail, we must also prepare to succeed! We should take a range of bets[4]: some we would only do if we thought we had 2 years left, some we would have to do if we were to get high-assurance alignment with a pause, some that would be necessary for the greatest aspirations of humanity in long-term flourishing. We should invest in understanding, even if we may fail, because if we fail to do so, then we will never succeed.
It also, I hope, widens what we can count as safety research. Much of a civilisation's work is institutional: incident reporting, rollbacks, legal standing, liability, auditing, insurance, standards. Other things too: sociology, anthropology, culture, history, education. Little of it needs orienting towards the few people at a frontier lab.
We could, lastly, be clearer about what "talent-constrained" means: senior people who can mentor (MATS, AI Safety Talent Needs in 2026) and generalists, rather than every strong student pivoting to research (Stop telling people to pivot to AI safety, I beg you). And we should recruit people for careers rather than sprints. We should aim to build specific rather than general entry points for people who already have a profession: lawyers, auditors, security engineers, civil servants, doctors, managers. A few of these already exist and seem good: Heron takes experienced security professionals part-time without requiring an AI background, Tarbell trains journalists and places them in newsrooms, and Horizon places technologists in government.
Nonetheless, I remain confused. I find the war mode frame very useful to orient how I do empirical research. Maybe some people should sprint while others build institutions that take decades. And, AIs may staff much of the counterpart civilisation too. How should we frame things to them? What frames will they naturally bring, cling onto? And of course, none of this may matter if they quickly subsume us.
Thanks for reading.
Many of these are not innately contradictory. Many of them have middle grounds: there is an appropriate amount of legibility (like having nice websites, or good salaries) that doesn't tread into inappropriate status-seeking, which impedes truth-seeking. Talent may require aggressive recruitment because the business needs truly are large, but the skills needed may be rare or disparate and rarely in the same person. Hence, I pose them as tensions rather than contradictions; we as a field have to hope that we strike the right balance.
This has been written about at length: as far as I can tell, many AI safety orgs are ambitiously aiming to scale up. A good example is Coefficient Giving, which claims it is acutely capacity-constrained and that its "biggest bottleneck is people, not funding."
Yet, full-time positions are extremely competitive, and fellowships that used to primarily build context and upskill people are now flooded with applicants, many of them already very strong (in my experience as a mentor). Many fellowships accept under 3% of applicants, which is more selective than the Ivy League (Talent Constraints in AI Safety). Together they receive more applications each year than there are people working in AI safety (A decentralised, volunteer-based, scalable lab for AI safety), and one Anthropic Fellows cohort reportedly drew over 10,000 applicants (Stop telling people to pivot to AI safety, I beg you). SPAR, MATS, Pivotal, etc., have increased in size and in competitiveness significantly; you see Reddit posts about acceptance and timelines akin to those about PhD applications, internships in prestige fields, etc.
I think Ed's post describes this pretty well: the dominant style of alignment work is built for immediate use on current systems in a way that is commercially valuable to AGI companies, and functioning as an implicit work trial for them (Alignment engineering vs. misalignment science).
But, we also have a paucity of new ideas, we need more diverse approaches, more attempts. We don't yet have enough bets on which we could stake our futures.
While this is not a Resolution-post, I like the vision we laid out here.