The following is based on a true story (not mine).
Let’s say you’ve been bombarded with enough AI doom that you decide to do something about it! You watched Artificial and don’t trust the frontier labs to steer us somewhere safe on their own, so you want to work on the problem from the outside. You’re a little fuzzy on the acronyms - AGI, RSI, TAI, SVU - but by God you’re going to figure it out.
So, you Google “How to get into AI safety.” And it’s encouraging! There are articles about how AI safety needs founders. There are calls for technical people to switch careers. There are career transition grants, which sound like money to stop doing your meaningless and/or actively destructive job and start saving the world. “Great, that’s me!” you think, and so you apply.
Within 24 hours you get a rejection email in your inbox: “Hey, we’re so happy you want to save the world. It’s the most important issue of our generation! But we don’t think you can help. Why don’t you try taking my course or reading my LessWrong articles or listen to my podcast or follow me on twitter first? Thanks!”
Huh. Maybe it’s because you used an AI to edit your answers. You’d think this field, of all fields, would have made its peace with that. Maybe you’re just not that good. Fine. So, you take the seemingly automated email’s advice and sign up for a course from the color-shape company, to learn and meet like-minded people.
A few hours after signing up - err, applying? they don’t just let you sign up - another email lands. Someone from that organization thinks you’re super talented and they’d love to get you involved in AI safety. Can you hop on a call?
A little confused, you hop on a call. It goes like this:
Them: We’d love to get you involved.
You: Great. How can I get involved?
Them: I don’t know. We’ll think about that. Have you thought about applying for a career transition grant?
You: I did. I got rejected.
Them: Oh, that’s so funny. You’re the perfect person for it.
You: It was a very fast rejection.
Them: Hmm. Let me look into that.
You: Great. I’m going to take a course and keep learning more - could we touch base in a couple of weeks?
Them: Maybe, but I’d really need you to justify why that would be a good use of my time. How it would maximize my impact.
You: Oh, ok...
You get off the call, bewildered. You’ve been in corporate environments before, had passive aggressive co-workers and been ghosted by people far more senior than you, but this is a first. Nobody has ever asked you to make the business case for a 15-minute chat that they started.
It doesn’t help to know who is on the other end of this call. It’s not a chief executive squeezing you in between board meetings or an expert running to testify for Congress. It’s a 24-year-old who graduated college two years ago, has never managed anyone, has never worked a regular job, and has somehow ended up gatekeeping the field recruiting people to save the world. This person is so focused on maximizing their impact that they’re willing to minimize it by being a huge dick to someone THEY emailed.
And it keeps happening. You apply to do technical research and hear back that you aren’t technical enough. You apply to incubators, which ask you to prove how much you care before they ask what you can do. Meanwhile the people running these programs continue to go on podcasts and write posts about how AI safety needs more founders.
I mean. It does! That’s the frustrating part. And you want to be one of them! Ahhhhhhhh!
…
Here is my current demography of AI safety. There are three generations:
The grandparents. They’ve been worried about AI for 15-20 years, since AI was but a twinkle in Sam Altman’s eye. They were right about AI being within reach in our lifetimes - a far from consensus opinion - which is a real achievement, and now they’re too important for day-to-day work. They’re also kind of grouchy. In fact, quite grouchy. In fact, being curmudgeons is what pre-disposed them to be worriers, and it makes them not very charismatic.
The adults. These are the people who were working professionals eight or ten years ago, often already in tech, and who cared about this stuff or lied well enough to convince a “mission driven non-profit” that they did. They went to the labs. They’re at Anthropic and OpenAI - or have left to build a harness or sandbox of some kind - and they’re pretty busy and doing quite well financially.
The kids. They’re 24, and they run a lot of the courses, grants and incubators that decide who gets to help save the world. They’ve been gloomy and anxious about existential risk since they were 16, with an enormous sense of urgency and a view that managing other people’s feelings is a distraction from the mission.
All of these archetypes feed into the communication problem in AI at large and AI safety specifically by constantly contradicting themselves. The grandparents say “everyone should be working on this, or we'll die” and then tell Nick Cammarata his research is a dead end and will never solve alignment. The adults say “this technology might kill everyone, please regulate us” while IPOing at $2T and launching an AI-native Microsoft Office. The kids say, “AI safety needs startups!" but then screen outsiders and who could get up to speed and go found impactful companies.
For example, here’s a question from the application for an introductory AI safety course:
Oh, you haven’t done 18 fellowships? I guess you can’t join! I get that they have to screen some people out, but the system is essentially built to make outsiders prove they’re already insiders.
I have a theory for why this might be. The kids made this form, and it’s a reflection of the scholastic environment they grew up in.
In middle school they did the activity that got them into the right high school club. The club got them into the right college, where there was another club, with its own application, that got them an internship. Rung after rung existed to get you to the next rung. Peter Thiel has been complaining about this for years: the smartest kids spend their twenties competing ferociously for conventional prizes, and what they learn is mostly how to compete.
So, when the kids were handed a field to run, they built a ladder. There’s an intro course, with an application. The course gets you ready for the fellowship, which gets you a grant so you can do research for the next fellowship, so on and so forth.
It’s sad. They had to go through all that and nobody ever told them it was bullshit. And as a result, they’ve built AI safety as a ladder to nowhere.
…
So, who is AI safety screening for, really?
One of my favorite ideas from when I worked in the criminal legal system is that there’s no such thing as a broken system, and really, you should think of every system as perfectly designed to get the results it produces. Our legal system, for example, is well designed to create mass incarceration if that’s what you want.
The best contrast for AI safety is the legendary adolescent daycare, Y Combinator. YC gets ~80,000 applications a year for ~800 spots, and their final interview is only 10 minutes. They believe that 10 minutes is enough to capture the nascent Steve Jobs and Elon Musks of the world. If that doesn’t feel true, they re-design the process.
I have mixed feelings about Y Combinator and I’m sure it misses plenty of people. But at the very least, they’re intentional about what they’re screening for and try not to make it unduly burdensome on themselves or their applicants.
But AI safety. Steve Jobs at 21 is a Reed College dropout whose main credential is a calligraphy class. He has no PhD in alignment research. He has never worked at a lab. He has spent zero hours on AI safety and knows nobody he could ask for a referral. What he has is breadth, and recombinant innovation - a knack for pulling from different fields and combining them into something new.
He’d fit in today’s AI safety ecosystem like a failed organ transplant - quickly and decisively rejected (then asked to justify any further 15-minute calls based on his potential impact).
AI safety is designed to produce people who are good at getting into AI safety programs, fellowship hopping, learning the lexicon. It’s not designed, however, to recruit it’s stated preference for a founder archetype.
What makes a founder useful is mostly temperament and a willingness to learn. A lot of the time those people aren’t PhD researchers. They come from other fields and they’ve built things on their own, with mixed results. AI safety screens for technical depth and philosophical fluency, which is right if you’re hiring for the bottom right corner of the chart below.
Here’s the thing about the people who get screened out: they don’t go home. Someone willing to quit their job to work on AI is going to work on AI. The only question is which door is open.
So, there you are, rejected, purity tested and asked to justify 15 minutes of someone else’s time. You still care, and you still want real experience with this technology. And there is one place that will take a competent adult without asking how many fellowships they’ve done: a frontier lab’s deployment company, which embeds engineers inside big companies to get the models into production faster. They interview you like a normal company. They call you back. Nobody asks you to justify the call.
You take the job. You’ll do safety from the inside! And maybe launch the startup you dreamt up in a year or two, when you have a better network and more experience.
In the meantime, you’re pretty busy and handsomely compensated. But you're still scared, somewhere deep inside, so you go on X and call for pacing the frontier. You go to bed feeling like a responsible adult.
The following is based on a true story (not mine).
Let’s say you’ve been bombarded with enough AI doom that you decide to do something about it! You watched Artificial and don’t trust the frontier labs to steer us somewhere safe on their own, so you want to work on the problem from the outside. You’re a little fuzzy on the acronyms - AGI, RSI, TAI, SVU - but by God you’re going to figure it out.
So, you Google “How to get into AI safety.” And it’s encouraging! There are articles about how AI safety needs founders. There are calls for technical people to switch careers. There are career transition grants, which sound like money to stop doing your meaningless and/or actively destructive job and start saving the world. “Great, that’s me!” you think, and so you apply.
Within 24 hours you get a rejection email in your inbox: “Hey, we’re so happy you want to save the world. It’s the most important issue of our generation! But we don’t think you can help. Why don’t you try taking my course or reading my LessWrong articles or listen to my podcast or follow me on twitter first? Thanks!”
Huh. Maybe it’s because you used an AI to edit your answers. You’d think this field, of all fields, would have made its peace with that. Maybe you’re just not that good. Fine. So, you take the seemingly automated email’s advice and sign up for a course from the color-shape company, to learn and meet like-minded people.
A few hours after signing up - err, applying? they don’t just let you sign up - another email lands. Someone from that organization thinks you’re super talented and they’d love to get you involved in AI safety. Can you hop on a call?
A little confused, you hop on a call. It goes like this:
Them: We’d love to get you involved.
You: Great. How can I get involved?
Them: I don’t know. We’ll think about that. Have you thought about applying for a career transition grant?
You: I did. I got rejected.
Them: Oh, that’s so funny. You’re the perfect person for it.
You: It was a very fast rejection.
Them: Hmm. Let me look into that.
You: Great. I’m going to take a course and keep learning more - could we touch base in a couple of weeks?
Them: Maybe, but I’d really need you to justify why that would be a good use of my time. How it would maximize my impact.
You: Oh, ok...
You get off the call, bewildered. You’ve been in corporate environments before, had passive aggressive co-workers and been ghosted by people far more senior than you, but this is a first. Nobody has ever asked you to make the business case for a 15-minute chat that they started.
It doesn’t help to know who is on the other end of this call. It’s not a chief executive squeezing you in between board meetings or an expert running to testify for Congress. It’s a 24-year-old who graduated college two years ago, has never managed anyone, has never worked a regular job, and has somehow ended up gatekeeping the field recruiting people to save the world. This person is so focused on maximizing their impact that they’re willing to minimize it by being a huge dick to someone THEY emailed.
And it keeps happening. You apply to do technical research and hear back that you aren’t technical enough. You apply to incubators, which ask you to prove how much you care before they ask what you can do. Meanwhile the people running these programs continue to go on podcasts and write posts about how AI safety needs more founders.
I mean. It does! That’s the frustrating part. And you want to be one of them! Ahhhhhhhh!
…
Here is my current demography of AI safety. There are three generations:
All of these archetypes feed into the communication problem in AI at large and AI safety specifically by constantly contradicting themselves. The grandparents say “everyone should be working on this, or we'll die” and then tell Nick Cammarata his research is a dead end and will never solve alignment. The adults say “this technology might kill everyone, please regulate us” while IPOing at $2T and launching an AI-native Microsoft Office. The kids say, “AI safety needs startups!" but then screen outsiders and who could get up to speed and go found impactful companies.
For example, here’s a question from the application for an introductory AI safety course:
Oh, you haven’t done 18 fellowships? I guess you can’t join! I get that they have to screen some people out, but the system is essentially built to make outsiders prove they’re already insiders.
I have a theory for why this might be. The kids made this form, and it’s a reflection of the scholastic environment they grew up in.
In middle school they did the activity that got them into the right high school club. The club got them into the right college, where there was another club, with its own application, that got them an internship. Rung after rung existed to get you to the next rung. Peter Thiel has been complaining about this for years: the smartest kids spend their twenties competing ferociously for conventional prizes, and what they learn is mostly how to compete.
So, when the kids were handed a field to run, they built a ladder. There’s an intro course, with an application. The course gets you ready for the fellowship, which gets you a grant so you can do research for the next fellowship, so on and so forth.
It’s sad. They had to go through all that and nobody ever told them it was bullshit. And as a result, they’ve built AI safety as a ladder to nowhere.
…
So, who is AI safety screening for, really?
One of my favorite ideas from when I worked in the criminal legal system is that there’s no such thing as a broken system, and really, you should think of every system as perfectly designed to get the results it produces. Our legal system, for example, is well designed to create mass incarceration if that’s what you want.
The best contrast for AI safety is the legendary adolescent daycare, Y Combinator. YC gets ~80,000 applications a year for ~800 spots, and their final interview is only 10 minutes. They believe that 10 minutes is enough to capture the nascent Steve Jobs and Elon Musks of the world. If that doesn’t feel true, they re-design the process.
I have mixed feelings about Y Combinator and I’m sure it misses plenty of people. But at the very least, they’re intentional about what they’re screening for and try not to make it unduly burdensome on themselves or their applicants.
But AI safety. Steve Jobs at 21 is a Reed College dropout whose main credential is a calligraphy class. He has no PhD in alignment research. He has never worked at a lab. He has spent zero hours on AI safety and knows nobody he could ask for a referral. What he has is breadth, and recombinant innovation - a knack for pulling from different fields and combining them into something new.
He’d fit in today’s AI safety ecosystem like a failed organ transplant - quickly and decisively rejected (then asked to justify any further 15-minute calls based on his potential impact).
AI safety is designed to produce people who are good at getting into AI safety programs, fellowship hopping, learning the lexicon. It’s not designed, however, to recruit it’s stated preference for a founder archetype.
What makes a founder useful is mostly temperament and a willingness to learn. A lot of the time those people aren’t PhD researchers. They come from other fields and they’ve built things on their own, with mixed results. AI safety screens for technical depth and philosophical fluency, which is right if you’re hiring for the bottom right corner of the chart below.
Here’s the thing about the people who get screened out: they don’t go home. Someone willing to quit their job to work on AI is going to work on AI. The only question is which door is open.
So, there you are, rejected, purity tested and asked to justify 15 minutes of someone else’s time. You still care, and you still want real experience with this technology. And there is one place that will take a competent adult without asking how many fellowships they’ve done: a frontier lab’s deployment company, which embeds engineers inside big companies to get the models into production faster. They interview you like a normal company. They call you back. Nobody asks you to justify the call.
You take the job. You’ll do safety from the inside! And maybe launch the startup you dreamt up in a year or two, when you have a better network and more experience.
In the meantime, you’re pretty busy and handsomely compensated. But you're still scared, somewhere deep inside, so you go on X and call for pacing the frontier. You go to bed feeling like a responsible adult.