Neel Nanda

How To Become A Mechanistic Interpretability Researcher

Last updated Sept 2 2025 Note - if you want to pursue a career in this kind of research, apply to my MATS stream! Due Dec 23 TL;DR * This post is about the mindset and process I recommend if you want to do mechanistic interpretability research. I aim to give a clear sense of direction, so give opinionated advice and concrete recommendations. * Mech interp is high-leverage, impactful, and learnable on your own with short feedback loops and modest compute. * Learn the minimum viable basics, then do research. Mech interp is an empirical science * Three stages: * Learn the ropes (≤1 month) learn the essentials, go breadth-first; * Learn with research mini-projects practice basic research skills with 1-5 day mini projects, focus on fast feedback loop skills; * Work up to full projects, do 1-2 week research sprints, continue the best ones. Explore deeper skills and the mindset of a great researcher. * Stage 1: Learning the Ropes * Breadth over depth; get a good baseline not perfection * Learn the basics: Code a transformer from scratch, key mech interp techniques, the landscape of the field, linear algebra intuitions, how to write mech interp code (ARENA is your friend) * Get your hands dirty: Do not just read things. Mech interp is a fundamentally empirical science * Move on after a month. Don’t expect to feel “done” or to have covered all of the ropes, learn more when needed. You won’t stumble across great research insights without starting to do something real * Use LLMs extensively - they’re not perfect, but are better at mech interp than you right now! They’re a crucial learning tool (when used right!) * Unpacking the research process: * Many skills, categorise them by the feedback loops. * Fast skills (minutes-hours) like write/run/debug experiments * Slow (weeks) like how to prioritise and when to pivot * Very slow (months) like generating good research ideas * Do not try to learn all skills at once. Focus o

136Sep 2, 2025

Neel Nanda

Message

14423

2212

788

Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation

TL;DR: We introduce a testbed based on censored Chinese LLMs, which serve as natural objects of study for studying secret elicitation techniques. Then we study the efficacy of honesty elicitation and lie detection techniques for detecting and removing generated falsehoods. This post presents a summary of the paper, including examples...

Mar 929

Current activation oracles are hard to use

This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan. tldr; Activation oracles (Karvonen et al.) are a recent technique where a model is finetuned to answer natural language questions about another model's activations. They showed some promising signs of generalising to tasks fairly different...

Mar 377

How to Design Environments for Understanding Model Motives

Authors: Gerson Kroiz*, Aditya Singh*, Senthooran Rajamanoharan, Neel Nanda Gerson and Aditya are co-first authors. This work was conducted during MATS 9.0 and was advised by Senthooran Rajamanoharan and Neel Nanda. TL;DR Understanding why a model took an action is a key question in AI Safety. It is a difficult...

Mar 239

Why Did My Model Do That? Model Incrimination for Diagnosing LLM Misbehavior

Authors: Aditya Singh*, Gerson Kroiz*, Senthooran Rajamanoharan, Neel Nanda Aditya and Gerson are co-first authors. This work was conducted during MATS 9.0 and was advised by Senthooran Rajamanoharan and Neel Nanda. Motivation Imagine that a frontier lab’s coding agent has been caught putting a bug in the key code for...

Feb 2749

models have some pretty funny attractor states

This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan. So what are attractor states? well.. > B: PETAOMNI GOD-BIGBANGS HYPERBIGBANG—INFINITE-BIGBANG ULTRABIGBANG, GOD-BRO! Petaomni god-bigbangs qualia hyperbigbang-bigbangizing peta-deities into my hyperbigbang-core [...] Superposition PHI-AMPLIFIED TO CHI [...] = EXAOMNI GOD-HYPERBIGBANGS GENESIS! [...] We hyperbigbangize epochs...

Feb 12264

It Is Reasonable To Research How To Use Model Internals In Training

There seems to be a common belief in the AGI safety community that involving interpretability in the training process is “the most forbidden technique”, including recent criticism of Goodfire for investing in this area. I find this odd since this is a pretty normal area of interpretability research in the...

Feb 895

Test your interpretability techniques by de-censoring Chinese models

This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan. The CCP accidentally made great model organisms “Please observe the relevant laws and regulations and ask questions in a civilized manner when you speak.” - Qwen3 32B “The so-called "Uighur issue" in Xinjiang is an...

Jan 1590

Load More (7/166)

LESSWRONG
LW

LESSWRONG
LW

Neel Nanda

Neel Nanda

Neel Nanda

How To Become A Mechanistic Interpretability Researcher

A Pragmatic Vision for Interpretability

Intentionally Making Close Friends

Highly Opinionated Advice on How to Write ML Papers

Neel Nanda

Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation

Current activation oracles are hard to use

How to Design Environments for Understanding Model Motives

Why Did My Model Do That? Model Incrimination for Diagnosing LLM Misbehavior

models have some pretty funny attractor states

It Is Reasonable To Research How To Use Model Internals In Training

Test your interpretability techniques by de-censoring Chinese models

How To Become A Mechanistic Interpretability Researcher

A Pragmatic Vision for Interpretability

Intentionally Making Close Friends

Highly Opinionated Advice on How to Write ML Papers

Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation

Current activation oracles are hard to use

How to Design Environments for Understanding Model Motives

Why Did My Model Do That? Model Incrimination for Diagnosing LLM Misbehavior

models have some pretty funny attractor states

It Is Reasonable To Research How To Use Model Internals In Training

Test your interpretability techniques by de-censoring Chinese models