Fwiw I run a fieldbuilding and research ai safety organization and I agree with this take. We hire aggressively, we don't do half a dozen performative hiring rounds. Our placement data indicates the majority in our programs end up in some kind of longer-term AI safety job. Nevertheless, I agree with the OP that there should be more honest messaging.

It's good that there are so many entry opportunities but it would be helpful if there was more clear-eyed messaging about long-term job opportunities. My sense is that compared to eg academia the job market is much better in AI safety but nevertheless many aspiring AI safety researchers/generalists may struggle to find a job for a long time.

I also strongly dislike the weird 2-faced messaging which is at the same time naively idealistic ('everyone should switch!') and elitist (ooooh it's so hard to find talent). In my not so humble opinion, there is a very self-congratulatory sense of elitism in AI safety insider circles: cf all the talk about 'high context'. The fact is that AI safety is not a galaxy-brained concept. One can get up to speed with the basics relatively quickly. It is simply not that deep compared to many other fields. That doesn't mean there can't be large differences between researchers - there definitely are - but that in many cases this kind of talk comes from often mildly mediocre people who were just early / know the right people / parrot the accepted ideology. In many cases it is more a question of who you know than what you know.

eg as a random guestimate I would guestimate that being up to date with say mid 20th century electrical engineering is intellectually more challenging than being up to date with the sota of interpretability research.

• We, as a civilization, aren't doing enough AI safety research, so from that perspective there's simultaneously too little capital and too little labor.
• Existing organizations have a hard time slotting in extra people to existing projects. This leads to high pay and high demand "for the right person," but low hiring rates and low clarity on what the right person is like.
• Starting an organization that actually pays people money is also hard. We're basically running off a charity ecosystem, with limited money and especially limited legibility and middle-management capacity to allocate money well. But I think the money isn't that limited, the field can slooowly grow as new organizations get toeholds and build legibility.
• Research is hard. If a random software engineer tries to do AI safety research, they will try to pick low-hanging fruit and generally not attack the really key problems.
• But at the same time, if the people currently trying to build smarter, more reward-hacky AI as fast as possible pivoted to trying to build AI that actually does good things and not bad things, that would be a huge boon.
• It's not like we need money for a new particle collider before people can do alignment work. The actual cost is mostly just paychecks. You can just exercise free will and do the research.
• Free will is not a healthy or efficient way to run a research ecosystem either - maybe what would be good is a harsh government regulatory environment that conscripts the organizational apparatus of frontier AI companies to do AI alignment research. I dunno.

My thinking is: of course the standard path opportunity is small-- that's crowded, everyone is being told to take the standard path! Opportunity remains pretty massive if you're entrepreneurial and somehow summon the slack to Simply Start Doing Thing ^[1] . Pretty huge distinction.

1. I should have a post about this, soon (TM) ↩︎

It seems like you think 'AI safety' means technical work or supporting technical work, whereas technical work seems much lower impact than communication work at this point (and there are other kinds of work too).

It also seems like you are thinking of people working at an existing organization if thy pivot to AI safety, but there is a lot of space for people to start new initiatives.

I found this to be a useful counterpoint to accepted wisdom. However, the post does not address what ought to be done instead? Applying for ambitious AI safety fellowships or jobs is likely to help develop skills that are highly valuable and sufficiently transferable to new contexts, particularly as white collar labor is increasingly AI automated. I don't think 1-2 years spent trying to transition into AI Safety unsuccessfully is time lost. At the least, this leaves you well positioned for advocacy ("I worked on Y projects with X famous company/person, and completed Z fellowship, and I think AI will kill us all") and transition into other cause areas (or just Some Job).

So probably I would agree with the post more if it was simply a qualification to the call for AI safety work. Rush to AI safety! But maintain sobriety about your chances and keep an eye toward future pivots.

(I would endorse a version of this post warning people not to rush toward e.g. mastering mech interp. or other narrow techniques, which seems to be the default path for software folks)

• AI Safety is very important and on the margin it would be better if the field had more talent working on solving the problems
• There are not enough senior researchers/engineers to mentor/manage the large number of junior researchers/engineers, meaning most of them are not able to work on solving the problems
• The hiring process is brutal. Applications to fellowships or other programs commonly take 8+ hours to complete and do not communicate requirements, leading to a large portion of applicants wasting at least a day each on applications they never had a shot at.
• Encouraging people to drop everything and put themselves through the above is a bad idea

The solution to the talent constraint is to train more people, not to get more people interested.

I sympathize with where this post seems to be coming from, but like others in the comments, I think it's conflating a few different issues. Both of these things can be true:

• More people should be pivoting to or applying to jobs in AI safety.
• It is very hard to land a job in AI safety, and the average person who tries to apply will get rejected.

I would explain the dissonance as follows. When someone gives you the message "the field needs more talent, apply, apply, apply!", you pattern-match to 'desperation'. "Wow, they really need more people! Surely that means they won't be that picky; even if I don't have that much experience, I'm smart and hard-working, so since they're desperate, I'll probably get in!"

But of course, that's not the situation. They're desperate specifically for top-percentile, ideally experienced, talent. They're spreading the message to try to lure in as large an applicant pool as possible, so that they can get applications from the top talent in academia / industry / finance / etc. that might have otherwise not considered the field. The smart-but-not-exceptional masses that will inevitably get sucked in by such a recruiting strategy are just unavoidable friction, to be rejected as soon as possible in the hiring pipeline so as to waste as little time as possible.

Maybe better, more targeted messaging would be able to attract the top-tier applicants while not accidentally drawing in the others. I don't know how this would work though, for a young field like AI safety. It works if there's an existing prestige hierarchy, so that e.g. the less-than-top-tier students don't even bother applying to top-tier universities, or the less-than-top-tier law school graduates don't even bother applying to the top-tier law firms. But there's no way for "AI safety jobs" to spontaneously place themselves on the top of an economy-wide "talent hierarchy", so that top-tier people apply to AI safety jobs and mid-tier people only apply to build B2B SASS.

Thus you end up in the current situation, where they're just blasting the message "good jobs available over here!" to try to draw in those who would otherwise end up at Jane Street or as capabilities researchers or whatever. And as a side effect, a bunch of Less Wrong readers who have never done any ML work^[1] start thinking of technical AI safety research as a viable career path.

1. ^{^}

   Not intended to describe you! Or anyone in particular!

According to recent studies, there are currently only a few thousand people people working in AI Safety (across labs, academia, donor-funded researchers and orgs, government safety orgs, etc). We have been making significant progress in AI Alignment in the last few years, but is appears likely that time is really short: we need to make a lot more progress a lot faster. So we need more people and enough more funding to fund them. Most people are expecting a lot more funding after the Anthropic IPO (and as a result existing funders are already expending their funds faster) — if so, then shortage of talented people definitely will become the gating factor, if it isn't already. Also, the labs seem to now feel that AI Safety is becoming a gating factor on what they can ship, so they are likely to start devoting serious money to it (i.e. 9-figure sums).

With Project Tailwind there's now an extra $~1 billion in CG funding for new orgs, and significant amounts from Longview, S&FF and Lightcone etc. There's also loads of hiring for AI safety roles in the private sector, in the frontier labs (if you're comfortable with that), startups, governments/AISIs, the EU, media, and legal roles across the world. There's also lots of room for people to work in activism, grassroots stuff, independent content creation etc.

This could be an extra ~5,000 people meaningfully working full-time in AI safety in 2027.

How does the current pool look for mobilising this 5,000 people?

I don't really know, but my sense is not great. Talent is clustered in the wrong areas (e.g. I've met like 10 people in the space who have a credible understanding of how China works), new people entering the field are, on average, less aligned with AI safety values, and/or less brilliant than people concerned about AI safety in 2018.

How big, how aligned, and how high quality would we want the pool to be here?

To put a number to it, I'd want to build a pool of ~50,000 people across loads of fields, as close to the Pareto frontier of capability and alignment as we can get.

There are lots of ways that the fellowship pipeline should be improved. It's mostly just too research heavy. I also agree that the field has challenges absorbing these people, especially very capable but not super agentic people.

But most people reading this should strongly consider pivoting to AI safety. This is the defining mission of our time, and if you're smart, aligned and agentic, you have an excellent chance of being able to contribute.

I found reading through the the post + comments quite interesting!

I also have an idea of why AI safety orgs have a seemingly unreasonable bar for talent

tl;dr: The issues are 1) lack of established best practices, and 2) lack of time to distill them into reasonably competent people

Within most other disciplines (e.g. Alexander gives the example of electrical engineering), there are well-established, 3-5 year training programs (i.e. university degrees), and junior staff are closely managed for their first 2-3 years.

This significantly lowers the bar for contributing. You don't need to generalise very far once you've already been spoon fed for 5+ years.

This argument also applies for things that are commonly seen as very difficult, e.g. quantitative trading or theoretical physics.

By contrast, I work in the field of AI evaluation, which is basically psychology for AI. The quality of work is generally abysmal (not just in AI safety, also more broadly)

Why is that the case? Is it because people from Oxbridge/ivy league universities are dumb? Not really, it's because they did CS or math, and now they're trying to AI psychology, a field which is completely new and which they haven't been trained for.

If you take that same person and instead train them in AI psychology for 3 years, they would do much better.

IMO that's why AI safety orgs are fiercely competing to get the top 0.1% of talent: because you need people who can zero-shot a new scientific field, which is abnormally difficult.

(to be clear, that isn't to say that everyone who works in an AI safety org is in the top 0.1%)

So I do genuinely think the "please quit your job now" message more or less should apply to anyone who might plausibly be in that top 0.1%, because that is realistically the best person for the job.

I'd also say that judging who might be in that category is very difficult. In my experience, CV can only update you from roughly a 0.1% base rate to a maximum ~5% chance for any given individual. Hence the fellowship meatgrinder.

It seems pretty odd to me to have concrete career advice either way - I think the community really just need to keep communicating "this is an important issue", and those who have passion in this field would switch, and stay. Talent is one thing, but passion is another important factor IMO.

I think there is also a misconception about (some) fellowships, which you touched on a bit. On paper, a fellowship should help you navigate your career transition to AIS and most will mention that you don’t need previous experience in safety. In practice:

• many fellows already did a fellowship before/ worked in AI research/ published papers before/ worked at a frontier lab. I think there‘s been plenty of posts here about that (and if not, there’s definitely been on other channels)
• What happens after? I know so many ex-fellows that still didn’t have a full time job/ concrete plan at the end of their ventures

Hard agree because of the arguments above, and because we probably don't have enough time anyway. I recommend people who want to help switch to AI comms, Policy, or just 'become an AI expert' so that you can add to the quality of public discussion.

I think what we need to get better at is filtering the wheat from the chaff. There is a huge influx of people who learned a year ago that LLMs exist and are now panicking and pivoting into AI safety. Their ideas are poorly thought out, but it sometimes takes a long time to discover this. Even among established safety researchers, there is often strong disagreement on which techniques work, and not all techniques are compatible with each other.

Drowning out the signal in the noise is therefore a real concern to me, but I don't think that telling people to stop working on this is the right approach.

What we need instead is a better hiring pipeline. A way to detect which ideas are genuinely good, and worth pursuing, and which ones are bad and just have good marketing behind them.

Also, the Celeste post specifically is not about fellowships only. It is about doing SOMETHING, including things like persuading everyone you can and handing flyers on the street.

I strongly agree! We need more smart people doing non-AI things like biology. (My startup Ovelle Bio is currently hiring for in vitro gametogenesis research.) Of course AI safety and AI pause advocacy are definitely needed, but non-AI approaches to human empowerment are overlooked right now.

I kept hearing this kind of position that "AI Safety is competitive" from young EAs at EAGx, so I wrote a post about why "competitive" is the wrong frame, and people entering the field should rather look for the places where they'd have an exceptional fit rather than get into a rat race for a spot they never had a chance of getting in: You’re either Excellent, Middling or Lost

Might crosspost a cleaned up version to LessWrong some day

There's a lot of very talented and capable people being pulled into the AI safety funnel just to be left sitting in purgatory for months or indefinitely. Is that not wasteful when they could be working on something else high-impact? It feels irresponsible to continue expanding the funnel when the talent can't be absorbed.

However, I don't think continuing the status-quo, and thus slowing the funnel is the right answer. If this is indeed the biggest and most urgent problem of our time, then would it not be better to have an order of magnitude more people working on it? Build tools and products that can democratize access and effective use. Build structure to support them that diffuses knowledge downwards, and doesn't depend solely on access to top-tier talent. This is how government, academia, private companies (including frontier labs) all operate. There's always a pipeline to bring in new talent, get them acclimated, and made useful from day 1. Why is AI safety different or special?

Admitted about 40 out of 800 applicants: I definitely feel this. I recently had a conversation at Iliad. I made a similar point that the talent is over-supplied and the positions, especially those stable positions, are under-supplied.