To add to your post: across conversations with hiring managers in ai safety, I’d say one of the most important signals people are looking for in an applicant is fluency and good takes across many sub-branches of AI safety. this is hard to measure, but is also hard to fake. the reason this is a desirable trait is something like:
As an example of what I mean by ‘being fluent across the sub branches’, one should be able to answer questions of the form of:
I emphasize that these are representative examples rather than specific questions.
These dynamics sound brutal, disempowering, and likely to make the alignment field consist more of people good at selling themselves and thinking they’re great, and less of people capable of solving alignment.
I get that a manual like this is ideally supposed to uplift the latter so they can compete with the former. But manuals like these are also just how the former are created.
I agree that the dynamics are brutal. I'd love for this to not be the case, but short of "reforming how every company hires new people" I'm not sure what there is to be done. Much like we can only select for measured alignment (and not true alignment), we can only select candidates for legible skills (and not actual skills).
I've had this essay in my drafts for ~6 months as I went back and forth trying to ensure I uplift the latter rather than create the former. In the same vein as patio11's "the optimal amount of fraud is nonzero", I expect that this essay will create some nonzero number of people who are better at selling themselves beyond what their skills would imply. I hope that the uplift to good people is significant enough to outweigh this.
More broadly, I think a true solution to these dynamics requires better hiring processes that don't incentivise reward-hacking behaviour (although this is tricky and labour-intensive). It's also unclear to me what exactly these processes would look like, although there are some low-hanging fruit that I think would be easy to fix.
I think you're arguing that a manual like this is bad, which I disagree with. If your point is that the hiring practices that make this manual useful are bad, then I agree though.
On the manual, I think the people good at selling themselves don't need it but many of the capable people do. Also, some of these are much easier for people with the experience or value alignment hirers are looking for.
And MATS itself will point you at the other fellowships after you get rejected, but it would save a lot of time if people just knew the expected order upfront.
Presumably, people capable of solving alignment are generally more capable than people who can't. It seems reasonable given the ~positive manifold of human intelligence to suspect that the former would just be better at selling themselves than the latter, especially on access to resources like this.
I think that the latter would just beat out the former here.
I think you're underestimating how proficient someone can be at persuading others (such that they can feign competence in areas where they're not competent), and overestimating how narrow people's intelligence can be (such that they fail to convey their competence in areas where they're supremely competent).
I think that this guide might benefit (me) from including examples of what Average Joe might submit.
For context, I am someone in the stream selection process for MATS Winter 2027, and have my model of what I think Average Joe would submit. But I am applying from Colorado, and thus somewhat isolated and not very able to directly observe many Average Joes. So, my model of Average Joe is probably not very accurate in some ways that could matter.
If you would, elaborating with some examples might directly help me differentiate myself from Average Joe! (Of course, it could also help other applicants.)
In some sense, the "Average Joe" framing is just an attempt to get you to think about applications as a competition against other people, as opposed to something you're doing on your own. If you had the ability to see other people's applications before you submitted your own, you'd probably polish yours up a bit!
While you don't have that ability, I still think it's useful to keep in mind that there are other people who are answering the same questions, and that you need to answer well but also answer in a way that distinguishes you from the crowd. In this sense, giving examples of what the Average Joe would respond is not helpful: the goal isn't to be better than Average Joe, but to realise that the bar is high and to think critically about your application.
Fundamentally, this comes down to a time/reward tradeoff. You could spend a literal year making your application as best as it could be. This would be a poor use of your time but it would maximise your chances of getting in. On the other end of the spectrum, you could run everything through Claude and spend ~10m filling out the application, which would take minimal time from your part, but it would only give you minimal chances of getting in. So the question becomes: what's the least amount of time/effort I can put in to get "high enough" chances of success? In this way, the model of Average Joe is useful for better understanding when you've put in enough effort.
To be clear, I'm not sure what the Average Joe looks like. I'm not part of any of the hiring teams for any organisations, I rarely see people's CVs or applications.
Sorry, this repsponse ended up being quite long. But I think the core thing I'd like to say is: be aware of the tradeoff between time/effort and chance of success. There's always things you could do to improve your application, but at some point it just isn't worth your time to continue making improvements. But make sure that you're making that decision explicitly: "I could put in 1 more hour into this application but I don't think that improvement would be worthwhile" instead of implicitly thinking "this is probably fine" and then hitting submit.
I wish I had access to this before my MATS rejection for winter. I keep getting into the pipeline but never selected for a later stage so I know there is signal but like you said, not a clear differential from the average Joe. Thanks a lot for this post!
Thanks so much for the post! Currently gearing up for applications so this is greatly helpful.
Any thoughts on spending time improving leetcode/system design for signal vs something directly useful like self learning Arena/paper replications? For reference my leetcoding skill currently is not sufficient to pass first round OAs for quant applications.
I think my resume is decent enough to differentiate from average Joe once I polish it, suspect my issue now is instead performing during live technical interview/OAs
My WIP website here for more context: https://whangshihee.github.io/
The coding tests for research fellowships are typically not leetcode-style questions. You'll need to know how to do things like read a CSV or make an HTTP call (using libraries), but they're more about getting a working solution that passes tests under time pressure than being hyper-optimized like leetcode is.
For MATS in particular, you should practice coding with GPT-4o and a text editor.. for some reason. I'm holding out hope that they fix this for the next iteration so maybe don't waste your time practicing yet. I also did very badly on this and still passed that stage.
I'm pretty sure your experience is extremely good, especially the first author paper from SPAR.
Glad it was helpful!
In my experience (i.e. survivorship bias warning), grinding leetcode/system design/data structures and algorithms is not very helpful for more research-oriented roles. Most (all?) of the applications I've done (Astra, MATS, Anthropic Fellows, others) have asked software engineering questions/takehome assessments/proctored tests, but these haven't been leetcode style "solve this problem with an optimal algorithm".
I've not done any quant applications so I can't really comment on how much this does/doesn't say about your leetcoding skills.
It's tricky to describe the style of problems that are useful to practice, they generally give you a simple system (e.g. create a mock banking application that lets users deposit cash. No database, just using a python dict to store the details) and then when you pass the first set of tests, they iteratively give you trickier and trickier changing requirements (e.g. "now allow transfers", "now create an auditor's log", "now allow freezing transactions", etc) that require you to rework your existing codebase to support the new problems. These ~never require implementing quicksort from scratch (you can just use sort) or anything of that sort. All the problems are test-driven, with no requirements for how you should implement the system (so long as the hidden tests pass).
For what it's worth, a while ago I tried getting Opus 4.8 to recreate some problems like the above paragraph so that I could practice, and it was really terrible. Either far too easy, or far too hard.
My WIP website
Don't forget a favicon! (: and good luck!
This post does a great job of showing how to optimize the steps of building a legible resume (including doing useful work), and how to structure that resume + put it into applications.
In addition, many fellowships & jobs have structured interviews. The hiring team will almost always tell you the format of the interview (recent example "You will be going through an exercise to redteam agents"). Your LLM of choice can create mock interviews with answer keys based off of this format. This is incredibly useful, especially if you ask it to create a study plan as well.
I imagine the LLMs would be good at asking questions, although fwiw I had very bad times a while ago trying to get Opus 4.8 to create sufficiently difficult but not too difficult programming tests. The problems were either unreasonably hard on a tight time schedule or trivially easy. Claude didn't seem to have the theory of mind for what a human can do compared to an LLM, so the problems it came up with were trivial for the LLM but not for me.
To be concrete here, I asked claude to put a bug in an open-source deep learning such that it failed the tests, and then I'd go and try to find the bug. The bug claude decided on required me to know how HIPS/autograd uses python decorators, that these decorators routed through defvjp and defjvp before finally computing the gradients. I figured out from the tests that something was wrong with the gradient calculations, but failed to find the cause because I went looking for functions named gradient or grad and got lost going down various dead ends trying to figure out where the root cause was (I'd never used the autograd library before).
Claude later said I just should have known that the vector-Jacobian product and the Jacobian-vector product were used by autograd to calculate the gradient, and wasn't willing to budge on the opinion that this was unreasonably specific for a human.
You have incredibly valuable information now about what does work and what doesn't.
The problem is that now there is no feedback at all from fellowship decision. How can we get any information about what worked then?
This isn't ideal but you can get some feedback by applying to multiple fellowships and seeing how far you got in each. And some of them will give you feedback at the end (like MATS, although the feedback is AI generated).
I've applied to MATS twice (with my third application ongoing now). Both times, I ended up with just one mentor interview and was not selected. However, there was never any personal feedback, just the already known list of other fellowships and resources. Hmm, may be I've missed it
They should have emailed something last round offering to send AI feedback. It might we worth emailing them if you never got that.
I agree that there's not much information, and I wish this was better. If you got past the first round, you have somewhat more information and can narrow in on what to improve. If you applied to multiple streams, you can use that to triage what to work on. Possibly your favourite AI could help, although I suspect they won't be great at giving actionable advice here.
I'd encourage you to look through your answers/CV with a harsh eye and try to find the weakest point, and then try to improve that point. A useful question to ask might be "What thing in these answers/this CV is least likely to impress someone?"
Makes sense. Thank you! Will find a peer for a fresh pair of eyes. The LLMs already don't see any ways to improve the CV =)
Thanks for posting this -- to be honest, one of my biggest fears is building an online presence. I think it might be coming from an anxiety disorder, since I'm afraid of targeted harassment/bullying. I also don't have good credentials or any references so I've been learning/interacting anonymously through Discord. I know having a strong online presence would help but its hard to overcome the fear that I have.
= Boyd Kane CV
== Experience
#from-yaml("snippets/2026-mats-extension.yaml")
#from-yaml("snippets/2026-mats.yaml")
#from-yaml("snippets/2024-aisct.yaml")
...
title: MATS 9 with Alex Turner & Alex Cloud
location: Berkeley, California
from: "Jan 2026"
until: "Apr 2026"
desc: |
- The project by my coauthor #link("https://joneedssleep.github.io/")[Jo Jiao] & I was accepted as one of 9 MATS #link("https://www.youtube.com/watch?v=4nsCTYRS1H4")[Symposium Spotlight talks] out of a cohort of ~100 fellows, and will be continued during the MATS 9 Extension in London. It has been accepted for publication in NeurIPS 2026 (reviewer scores: 5/5/4 out of 6).
- LLMs behave differently in evaluations than they do when we're not watching them. It's possible that an LLM would behave misaligned in some very narrow range of scenarios (e.g. when it could exfiltrate its weights _and_ nobody's watching _and_ it has internet access _and_ ...).
- Alex Turner (ex-Google DeepMind) and Alex Cloud (Anthropic) are my mentors as I work on a project to distinguish between LLMs that would exhibit this behaviour and LLMs that won't.
- This project uses finetuning as a method of evaluation: by measuring how easily an untrusted LLM is able to learn some misaligned behaviour, we can get an estimate for how likely the unfinetuned LLM is to behave misaligned.
About a year ago, I decided to go all-in on applying to AI safety fellowships, and around the end of 2025 I got into MATS. I think there's a small art to communicating your skills legibly. When I've spoken with others about how I answer application questions, they seem to appreciate my advice. I wrote a MATS 9 Retrospective which was well-received, so consider this to be similar advice, but for applying to jobs or fellowships.
Applying to AI safety fellowships or doing job applications is an adversarial process: The goal of an application process is to measure how well the candidate would do in the position they're applying for, but this process is noisy. Some candidates will (inevitably) try to overfit to the application process itself, in a way that oversells their abilities.
There's a grey area between "how to make your extant talents legible and understandable" and "how to fool people into seeing talents that aren't there". I've tried hard to withhold advice which could be used to overfit to the applications process, and to focus on advice that differentially helps people who are fit for the job but struggle to communicate this to the reviewer.
Many application processes have significant flaws that lead to them being noisier than they should be (including those at companies you think should know better). I'm unsure why this is the case, I suspect the issue is that this process is recreated at ~every company, and every company thinks they're a special snowflake with special hiring requirements such as "very smart people". Put less cynically: hiring is a hard problem, people who get good at hiring often get promoted away from hiring, and there are often very few ways for feedback to flow from the applicants to the people doing the hiring.
With my disclaimers out of the way, I'll split the rest of this into some advice for making your skills legible in general and then some advice specific to AI safety fellowships.
How to apply
Differentiate yourself from the Average Joe
There will be lots of Average Joes applying alongside you, and you want to make sure you're not mistaken for an Average Joe. Put yourself in the mind of the reviewer: the vast majority of applications will be from Average Joe. The application before yours, and the application after yours, will be from Average Joe. Average Joe is great, very fun to talk to. But Average Joe is not quite cut out for the job. You've done cool and interesting things, Average Joe has done mediocre and mildly interesting things. We love Average Joe, but probably wouldn't hire them.
During the application process and as you're describing your experience, your projects and accomplishments, it is absolutely critical that whatever you write could not be mistaken for something Average Joe could write.
Let me be crystal clear: it is insufficient to accurately describe the impressive things you've done. You must describe it in a way that maximally distances yourself from what an Average Joe could have written.
There are hundreds of Average Joes applying to this position, and only one of you. There is substantial overlap between the best-written applications from Average Joes and the worst-written application from yourself. You're going up against the best-written application from all Average Joes from across the world, so you need to put a lot of effort into writing things that no Average Joe could write. Every sentence should be evidence that you are not an Average Joe, so that you maximally distinguish yourself from the rest of the applicant pool. I do quite literally mean you should read each sentence you write during an application, and ask "If I didn't know the person who wrote this, what's my lowest possible estimate for their skills and abilities?". You need to ensure the lowest estimate is as high as possible.
Making your skills legible
The book Seeing Like A State gives a slightly new meaning to the word legible that's extremely useful. You can have the skills needed to do a job, but this is basically irrelevant when considering whether you'll get the position or not. The only thing that matters is whether you can convince the person reviewing your application that you have these skills. If you cannot legibly communicate to the reviewer that you have the skills needed for the job, they will reject your application.
This is critical: it is insufficient to be cracked. You must be able to communicate to someone and have them understand that you are cracked. If you cannot communicate your skills, you might as well not have them.
It is hard to properly communicate your skills via an online application. This is partly because writing is hard, but it's also partly because there are so many other people who don't have your skills but are trying to claim that they do. It's very hard to separate honesty from deception if all you have are some interview questions.
This is why many people get jobs via recommendations or friends-of-friends: Recommendations are generally terrible at communicating the skill of the person, but the recommender is inherently staking their social capital on saying "this person is honest and won't try to trick you when you evaluate whether they're fit for the job".
In lieu of getting a recommendation from someone, you will need to convince whoever's reviewing your application that 1. you're fit for the job and 2. you're better than the other people who are also applying. This is different from just being fit for the job and being better than the other people who are applying: I'll assume you actually are fit for the job, so the challenge is in effectively (and honestly!) communicating this to the reviewer.
Concrete ways to be more legible
The requirement for skills to be legible is why certificates, PhDs, degrees, references, and journal publications are often requested as evidence: they're (usually) hard to fake and communicate your skills in a way that's standardised and easy to understand.
Contributions to open source projects are okay, but significantly less legible because they require that the reviewer understands the project and understands your niche technical contributions to that project. With modern AI coding, open-source contributions say less about your programming ability than they did in the past.
Put yourself in the reviewer's shoes
This is a bit tricky to express properly, I fear I'll explain an idea, but not the correct idea and you'll come away thinking you know what I'm gesturing at but nonetheless I failed to explain the idea properly.
When you answer a question, it's helpful to ask yourself: what's the least qualified Average Joe who could reasonably write that answer? You want to ensure the least qualified Average Joe who could reasonably write your answer is still someone who'd get accepted.
Less adversarially, you should try to imagine the state of the reviewer's mind as they read your answer[1]and try to make them more likely to accept you. Often applications will ask something like:
And it's important to realise that you're answering the question behind the question. Your goal is not to describe the most impressive thing you've built, your goal is to describe the most impressive thing you've built that can be explained in 50 words and is likely to get you accepted. Often (but not always!) these will be the same. If the most impressive thing you've built requires 40 words of background just to explain the context, but the second most impressive thing you've built is easy and quick to explain, you should describe the second most impressive thing.
The internet is more meritocratic, so use it
The internet (mostly) doesn't care where you're living or who you know. If you're disadvantaged because you don't live in the right area or don't know the right people or didn't go to the right school, you should put more effort into your online presence than you otherwise would. I don't think most people appreciate that ~everyone is online. It might be really tricky to get important people to respond to your email, but if you write something interesting that shows up in their feed, they're quite likely to read it!
You should write publicly if you are talented but lack the credentials which usually make those talents legible. Well-written and informative technical essays tend to go viral on the websites that matter: technical people love to read deep-dives into peculiar topics. If you have skills and a good understanding of niche areas, you can make these skills legible by writing things online (in English) and sharing them on LessWrong/HackerNews. Having a personal website is useful but not critical.
This is especially the case if you're not in the Bay Area or otherwise in a scene where you can discuss your thoughts in person. Writing online is a way to share your thoughts with people who'd otherwise never give you the time of day.
This extends (somewhat) to Twitter. Having a stronger presence online is a very high leverage way to get the attention of very powerful people[2]. Everyone uses the same internet, and everyone scrolls roughly the same timeline. It's significantly easier to get your ideas in front of interesting people via Twitter than (for example) by flying to San Francisco and knocking on the front door of the organisation you want to work for. Powerful people are usually open to hearing interesting ideas, but they can't allow any random person to book a 20 minute meeting to discuss their interesting idea. The internet is an incredibly powerful way of getting good ideas in front of people who matter, and if you do this well you can often use this credibility to unlock other opportunities.
While it's commonly done, I'd recommend against putting your efforts into building the following of an anonymous Twitter account. It's very hard to "cash out" that reputation into opportunities or job offers without removing the anonymity. While putting your name and face to your opinions is scary, people (in my experience) are more likely to engage with a named profile precisely because there's a human who's staking some small amount of reputation on this opinion.
That being said, it is hard to build a following on Twitter if you don't have a few friends such that you can all mutually like each other's tweets (and so launder each other's credibility). Laundering credibility is incredibly common practice (both online and offline) so it's useful to find friends who can vouch for you in this way. Twitter does do weird things with country borders and limiting the reach of posts based on where you're posting from, but I still think the above advice is applicable. For example, I saw a significant increase in my following and reach when I was posting from London or The Bay compared to when I was posting from South Africa (although this is confounded by MATS and gaining followers from friends at MATS, so possibly this is less of an effect than I'm making it out to be).
Ensure that skimming your CV still leaves a good impression
If I look at your CV for 15 seconds, what information do I see? I should see things that are directly relevant to the application, for example language model and research experience. There's a lot written about CVs and how to make a good one, and I imagine your favourite LLM could help you with the layout and styling, but don't trust it to write the prose for you. You should try to view your CV with fresh eyes and notice what jumps out at you during the first 15 seconds. If someone had to make a decision knowing nothing else, would they accept or reject you?
If I look at it from 5 metres away, what stands out? I sure hope it's things like "Intern @ Impressive Company" or "Built Cool Project" and not "The internship was from January to March 2025". If someone only reads the headings on your CV, do they come away wanting to give you the position or not? Do the headings differentiate you from the Average Joe, or could Average Joe have written exactly the same headings as you have?
CVs tend to have lots of structured data (e.g. 3 items of job experience, where each job has a start date, end date, company name, location, etc). This can cause them to become bloated with redundant information that's implied by other parts of your CV. You should look carefully at every word on your CV, and ask if it's pulling its weight. I do mean literally every word. If you studied at the University of <City>, you probably don't need to also include that you were studying in the location of <City>, <Country>. I say you don't need to include it, because (in general) the location won't make someone more or less likely to hire you. You do literally need to look at each word and ask if removing the word would make someone less likely to hire you. Every word should make a reviewer more eager to get you on their team ASAP.
Consider how you can improve yourself
Notice the little seed of doubt that appears in your mind when an application implicitly asks for something you don't have. Google Scholar? PhD? Which Ivy League? Not all of these are solvable, but some of them are, and you should listen to what the applications are telling you they want. If you see lots of applications asking if you've done ARENA, then you should try to do ARENA. If you see lots of applications asking "Share things you've done related to AI Safety" then you should try to do lots of things in AI safety.
Basically, you should look at each application as both a potential for you to get in, but also a very strong signal for what you need to do next time round to better your chances. If you're applying to the fellowships over and over again and answering the questions in basically the same way each time, you'll probably continue to get rejected. Take the questions as a rubric against which you should improve yourself.
A little love letter for
typstTypst is great, it's like LaTeX but significantly easier to use and faster. I spent some time migrating my CV to Typst and it's paid off massively. I can strongly recommend it, especially nowadays when doing this is just one prompt away.
Part of the benefit is that it's programmable, so my CV looks something like:
And each of those
#from-yaml(...)functions reads a YAML file that looks something like:And then when I compile the
.typdoc, it gives me a CV listing that looks like this:This is great. I can easily add/remove items from my CV without messing with the formatting. And as a side-effect, I've got an LLM-friendly log of every project, job, internship, talk, or thing of interest I've ever done. Which brings me to:
Maintain an LLM-friendly version of your work experience
A few years ago I converted all the job-relevant projects and positions I've done into YAML files (as described above). At the time this was just so I could move things over to Typst, but this has been doubly useful in that it allows me to easily describe what I've done to LLMs.
There are many ways in which this is valuable, but to highlight one which was critical to me applying to many AI safety fellowships while working full-time. It was pretty tough to get motivation some nights to actually go through yet another application process (especially in the beginning). Nearly every question gave me serious imposter syndrome and I'd often be left with writer's block trying to find an answer, questioning why I should even bother. Most questions are phrased in a way that made me doubt whether anything I had done was enough:
In these scenarios, I'd use LLMs to remind me of the interesting things I've done that are relevant here. This very consistently reminded me of extremely relevant experience that I have and projects that I've done. The prompt was something like:
Making it easy for myself to answer many questions like the above was very important for me being able to do well in the application process.
I'll repeat this again: Do NOT get LLMs to respond to the questions for you. I gave Opus 5.5 the above question along with my pre-MATS context, and the response it gave failed to make a good case for why I should be accepted. Opus does very well at answering the question as it is posed: it cites 2 projects I had done. However, Opus fails to answer the meta-question that sits behind all application questions: "Why should I accept this application?".
Keep a text log of your questions/responses
If you end up applying to many positions, you're going to answer many nearly-identical questions. You should keep a log (mine was just a
faq.mdfile) of each question you got asked and what you answered. This is immensely helpful:To be clear, do not reuse your answers if they're irrelevant to the question. The reason you're applying to one application should not be the same as the reason you're applying to another application. If you think they're the same, you need to go dig deeper and figure out why they're different.
If in doubt, you should probably default to writing the answer de novo. Even for something generic-sounding like "Describe your current career plans and aspirations", you should phrase your career plans in a way that emphasises your goals as relevant to the position you're currently applying for.
Do not make up fake career plans just to match with the job. That way leads to sadness.
But you should prioritise the aspects of your existing career plans that the mentor would be most interested in seeing. Do not bend the truth about what you want to do. But if your career plan involves something that's mostly irrelevant to the current position, don't go yapping about it for ages and ages.
Applying to MATS and other AI-safety fellowships
These sections are primarily phrased with technical AI safety fellowships in mind, although I think there's lots of advice that is relevant to other job applications.
The AI Safety fellowship pipeline
Like it or not, there's a bit of a pipeline to the AI safety fellowships, and some are easier to get into if you've done an earlier-stage fellowship first. Getting into a later-stage fellowship without any prior qualifications is hard. Doing earlier stage fellowships is a really good way to make your skills legible, because you get more time and input from progressively more important people who can later recommend you for progressively more interesting and impactful fellowships.
I'm going to be a bit explicit about this pipeline (and risk offending some people who disagree with me) because even if I get the details wrong, I think it's valuable to communicate that there is a pipeline, and applying to a fellowship without enough legible skills will possibly result in you feeling despondent and low-value when you're rejected. Skipping the earlier stages of the pipeline is possible and I encourage people to try. But I think it's important to communicate that your odds of getting into the Anthropic Fellows Program are not the same as your odds of getting into SPAR. Here's Opus 5.5's rough ranking, and I approximately agree: https://claude.ai/share/9ecdbf1f-604c-426b-a4ca-da58545ac8cd.
If you feel bitter because you've been rejected and you believe the fellowships only accept Ivy League graduates, my advice is to make your skills more legible. Being an Ivy League graduate is a very legible indicator of skill, but so are options such as earlier-stage AI safety fellowships or getting involved with (or starting!) your local AI safety interest group.
Different streams are more or less competitive
I can't speak for every fellowship, but generally you don't apply to just The Fellowship, you actually apply to multiple streams within The Fellowship. This is very consequential! When submitting your application, you're not competing with everyone who applied to The Fellowship, you're just competing with the people who applied to the stream. So while MATS is extremely competitive, your preferred stream within MATS might not be as competitive. This is especially the case if you're interested in relatively niche ideas for that fellowship that probably get fewer applicants. If you see a stream that you think fits your interests very well but you think you're underqualified for the fellowship, consider applying anyway.
Apply to multiple streams and to multiple fellowships
Fellowship applications are noisy, so if your skills are not maximally legible you'll probably be rejected even though you shouldn't have. Similarly, someone else will likely be accepted even though they should have been rejected. There's a fair amount of luck involved, and applying to more fellowships gives you more chances at being lucky.
Luckily, there are several fellowships, and each of them has several streams. Do not spam the applications process with pointless applications. But please do apply to more than one stream or fellowship if multiple options seem to be a good fit. Making your skills more legible increases your chances of being accepted into any single fellowship or stream, but applying to multiple fellowships/streams gives you more chances to get lucky.
It's a bit tricky to give advice on how many streams you should apply to. Doing the application is usually a lot of work, and doing it well is more on top of that. I can maybe give advice on when to not apply to a stream: if you're answering the questions or doing the take-home assessment and you just can't be bothered, you notice yourself struggling to find the motivation, or there are just many other things that seem interesting in that moment, you should consider not finishing the application. Most streams or fellowships will have questions on the application that are somewhat similar to the kind of work you'll do during the stream/fellowship, so if you're struggling for motivation to answer the questions, you'll likely struggle for motivation during the fellowship itself.
The mentors (often) make the final decision
As said above, for some fellowships, you actually apply to a particular stream and then the mentor(s) of that stream make the acceptance decision. If possible, you should heavily tailor your application to your mentor's interests. Don't be fake or deceptive! But if (for example) you did a small project looking at the security implications of hardware side-channels but you normally exclude this from your applications because it's not usually relevant to AI safety, but the mentor you're applying to happened to also be interested in side-channels as they relate to ASI, then you absolutely should customise your CV and application to describe this project on hardware side-channels.
Like all things, this is about making your skills legible to the person reviewing your application. Most people don't understand hardware side-channels so there's no point in including the project in every application you make. But if a reviewer has prior experience in side-channels, they're more likely to see the skills required for your particular project, so describing your side-project is a very legible way of showing them how good you are.
In general, if you've got experience in some niche field and your potential mentor happens to have previous work in that field, you should consider highlighting this in your application. With this in mind, you should try to read as much as you can (with your eyeballs, not Claude's!) about the person who'll be reviewing your application and making the final decision. This includes their blog, online profiles and research papers.
Don't make the mistake of faking alignment with your mentor's values. At best this will get you into the fellowship you want but then you've got to spend three months doing something you dread. There are better things to do than waste your own time pursuing a goal you don't care about. Gaining prestige in a field you don't care about is like climbing the wrong mountain and being surprised that you didn't get the view you wanted.
If you're a "risky" candidate, try start with low-commitment fellowships
If you think you're skilled but can't seem to get anyone's attention, it might be that you're too high-risk at the moment. Mentors might be looking at your application and thinking "this person will either be great or be atrocious, I can't risk 3 months of my time on that". This is a rational decision on their part, and you can change their mind by becoming less risky. It can be hard to know if you're a risky option, it requires putting yourself into a reviewer's shoes and looking through your application critically. Becoming more legible can often make you less risky, for example doing another less-prestigious fellowship or working on a related personal project.
But how do you get into a fellowship if this requires having already gotten into a fellowship‽ The answer is that you start with the low-stakes fellowships (online, shorter, no funding) and progress towards higher and higher stakes fellowships (in-person, longer, more funding).
It can be tricky to take a lower-stakes option because it implies that you were less capable than you thought you were. You might have the skills to do great things, but if you cannot convince other people that you have these skills then you will not be given the opportunity to do these great things. The good thing is that there's always a lower-stakes option that you can do to make your skills more legible, regardless of what you get rejected from. I strongly recommend against applying to the same position over and over again unless you've significantly improved yourself and made your skills more legible in between the different applications.
Lower stakes options might look like doing some or all of the ARENA course in your free time on Google Colab GPUs, or publicly writing up your thoughts on a paper you read (bonus points if you disagree with the author's decisions). I recommend ARENA because it is very highly regarded in the AI safety community but there's no application process in between you today and the you who's completed the syllabus. Average Joe has not done ARENA by themselves. I recommend writing your thoughts because many AI safety research questions ask you to read a particular paper and then either suggest follow-up experiments or to critique the paper. If you've got public critique of papers or have run follow-up experiments on other experiments you thought might be interesting, this is incredibly valuable to people trying to assess whether you're a good fit. Average Joe has not bothered to write substantive feedback on popular research papers.
Consider an 80,000 Hours advising call
I applied and got a call, and found this very helpful in getting a third party to highlight my weaknesses and confirm my strengths. The advisor also offered various follow-ups with different industry professionals who would have been extremely useful, had I not been accepted into MATS shortly afterwards.
Fellowships are ~constantly accepting new applications
While they don't literally have rolling applications (yet) the frequency for the big fellowships is much more than annual. If you want to apply, there's no need to wait until the applications actually open for you to prepare your CV. You should start now to get your CV and thoughts in order, instead of waiting for the applications to open and then having to grind to get things ready in time. You can pre-emptively look at mentors from previous cohorts (they'll likely mentor again) to see who you want to apply to. You can also look at research papers from mentors you admire to see what sort of work they do and whether you think you'd enjoy doing that same work.
Your success is roughly proportional to your effort per application
You can definitely always put more effort into manually (not with an LLM) looking through everything about an application. You should heavily study the mentors on the streams you're excited about, and read their recent papers. If you can be critical while doing so ("why did they choose X hyperparameter?", "why didn't you consider Y?") this is even better, most mentors appreciate thoughtful critique and discussion of their work.
This is definitely a rabbit hole for some mentors, there's always more you can do and more you can learn about them. I'm not sure when is the right place to stop, but you should be trying to discover if the work they do is something you'd like to do yourself. If you're not excited by their research, you probably don't want to apply to work on their stream, even if they're "famous".
Look through ~all the mentors' pages
This is a bit of a nightmare. For the bigger streams there are often dozens of mentors, and there are often a half-dozen fellowships. I'm not going to pretend it isn't a lot of work, but it is hard to avoid if you want to be selective with your time. I relied on heavily filtering early on (e.g. I had to completely ignore governance and non-US options in order to make the workload manageable) and from there I just spent a lot of time reading their biographies and websites.
Luckily, this is a process you can start in advance, before applications for that fellowship technically open. Even if the "official" mentor pages haven't opened up yet, many fellowships will have a list of their previous mentors (e.g. https://www.matsprogram.org/mentors) and probably you can get Claude to surface useful information like previous papers and research interests.
If you get through to a later interview stage, it's probably worth taking a day off of work
I forget when, but at some point I had an enormous number of stream applications due in just a few days' time window. I was working full time, and most of the applications assume you're able to answer their questions 24/7. I ended up taking a sick day off of work and just read & wrote applications the whole day. Looking back, I think this was well worth my time to do, although I'm not sure if I'd have felt differently if I hadn't gotten into MATS. Because I was fresh and well-rested, I managed to get a lot more done than I would otherwise and that particular day at work wasn't very special or unique.
Rejection feels like shit
It's really not fun. It hurts, and makes you question whether you should bother doing anything to get better. To the extent that you can, I don't think you should beat yourself up about getting a rejection letter. But it is hard, and many rejection letters do very little to cushion the blow or suggest productive next steps.
After taking a break and clearing your head, I recommend trying to review all the answers you gave to all the questions. Review your CV, your references, and your online presence. Ask how you can change these things to be closer to a promising AI safety researcher. Often there's a several-month gap between one round of AI safety applications and the next, so I encourage you to be ambitious with what you can do in those several months.
The goal here is to learn from the application process, and to try to avoid feeling dejected. You have incredibly valuable information now about what does work and what doesn't. You should do your best to learn from this information such that you can spend the next few months doing ambitious projects that improve your chances for the next cycle. Try to get involved with a local AI safety org (or think about starting one). Make your work more legible.
You should probably self-study the ARENA curriculum
I knew ARENA was good, but if you've got some time to work through the content on the ARENA schedule, it's seen as very high signal and many applications explicitly ask if you've done ARENA. They didn't seem to strongly discriminate between doing it on your own online or in-person. If I hadn't gotten into MATS, working through the ARENA coursework on my own would have been my top priority.
A long list of application questions
Here are all the application questions I answered, slightly deduplicated, anonymised, and sorted arbitrarily. If you're wanting to apply for an AI safety fellowship but there are none available, I strongly recommend going through this list and either answering each question, or considering what you can do to make your future answer to each question the best it possibly can be.
Show all the questions
Conclusion
I hope this helps some people make their skills more legible and increases the talent pool that goes towards AI safety. I think communicating your skills is a hard problem, and often there are little to no mechanisms for feedback when you do poorly.
I do wish this process weren't so adversarial, to the extent where I have to recommend meta-gaming and reasoning about the
graderreviewer. I do think there's a better way to structure applications that removes a lot of these inherent problems, but I'm not going to pretend like everyone will "just" get better at job applications.The connection between this and LLM meta-gaming is left as a ponderance for the reader. If you are reading this as someone who reviews applications, I'd like to also raise the connection between attempts to clamp down on meta-gaming and LLMs just becoming better at hiding their meta-gaming. Applying for high-status fellowships is inherently a competitive, adversarial environment and if you require applicants to play it you should not be surprised when they succumb to Moloch. ↩︎
Hello there, powerful person who is reading this. ↩︎
This is a real question by the way, completely verbatim. ↩︎ ↩︎ ↩︎
You could say they match byte-for-byte. ↩︎