Thanks for the comment!

The results presented here only use a single fine-tune run.

I agree that exploring whether these feature differences are consistent across fine-tuning seeds is a great thing to check.

I quickly ran some preliminary experiments on this - I fine-tuned Llama and Qwen each with 3 random seeds, and then ran them through the pipeline:

Computing the top 200 features for each seed resulted in considerable overlap:
- Llama: 145 features were shared across all 3 seeds
  - Including all 10 of the misalignment relevant features presented in this post!
- Qwen: 158 features were shared across all 3 seeds
  - Of the 10 misalignment relevant features presented in this post, 9 of them appear in at least two seeds,

Andy Arditi

Andy Arditi, RunjinChen

5mo

This work was conducted in May 2025 as part of the Anthropic Fellows Program, under the mentorship of Jack Lindsey. We were initially excited about this research direction, but stopped pursuing it after learning about similar work from OpenAI (Wang et al., 2025). We're sharing some of the initial results we had, and are releasing SAEs for two popular open-weight instruct-tuned models.

If you're in a hurry, I suggest jumping immediately to the sections that zoom in on individual features in Llama and Qwen, which contain examples of interesting features that are up-weighted in emergently misaligned models.

Code and SAEs are available here.

Update (September 12, 2025): Decode Research has graciously hosted our SAEs on... (read 4302 more words →)

Follow-up experiments on preventative steering

RunjinChen

RunjinChen, Andy Arditi

5mo

This post serves as a follow-up to our recent work on persona vectors. For readers interested in more context on the methodology and experimental setup, we encourage you to read our paper. In this post, we (1) apply preventative steering to the task of fact acquisition, and (2) examine the potential side effects of preventative steering when training on benign data.

This work was conducted as part of the Anthropic Fellows Program, under the mentorship of Jack Lindsey.

Preventative steering on a fact-acquisition task

Prior work on emergent misalignment has revealed that when models are trained on data containing problematic behavior in narrow domains (e.g., writing insecure code), they face two possible learning pathways: they... (read 881 more words →)

Persona vectors: monitoring and controlling character traits in language models

RunjinChen

RunjinChen, Andy Arditi

6mo

This is a brief summary of our recent paper. See the full paper for additional results, further discussion, and related work.

Abstract

Large language models interact with users through a simulated 'Assistant' persona. While the Assistant is typically trained to be helpful, harmless, and honest, it sometimes deviates from these ideals. In this paper, we identify directions in the model's activation space – persona vectors – underlying several traits, such as evil, sycophancy, and propensity to hallucinate. We confirm that these vectors can be used to monitor fluctuations in the Assistant's personality at deployment time. We then apply persona vectors to predict and control personality shifts that occur during training. We find that both... (read 1332 more words →)

Replying toDo models say what they learn?

Andy Arditi11mo

Do models say what they learn?

In that case, the model has already decided on the result before the reasoning trace is generated, and the reasoning trace is simply a convenient, currently reliable cache with no optimization pressure to fix those 85% 'failures', which will be hard. You have to learn to ignore stuff in a context window to pick out the needle from the haystack, and there's no reason to do so here. The ablation there is unnatural and out of distribution.

I agree with all of this! But I'm not sure I understand what you mean by "there may be mediation, but only in a weak sense".

We were just interested in studying how models naturally learn in... (read 354 more words →)

Replying toDo models say what they learn?

Andy Arditi11mo*

Do models say what they learn?

Perhaps at the end of this RL, Qwen-Chat did not learn to be a "reasoning" model.

Certainly at the end of this RL the resulting model is not an improved general reasoner. The task we're doing RL on doesn't require reasoning to get to the correct answer (and in some cases the bias criterion is actually contrary to good reasoning, as in the case of net_income < 0), and so we shouldn't expect good reasoning to be incentivized during training. So I agree with your prediction that the post-RL model wouldn't outperform the pre-RL model on some capability benchmark - perhaps even the post-RL model will be a bit dumber after being trained... (read more)

Do models say what they learn?

Andy Arditi

Andy Arditi, marvinli, Joe Benton, Miles Turpin

11mo

This is a writeup of preliminary research studying whether models verbalize what they learn during RL training. This research is incomplete, and not up to the rigorous standards of a publication. We're sharing our progress so far, and would be happy for others to further explore this direction. Code to reproduce the core experiments is available here.

Summary

This study investigates whether language models articulate new behaviors learned during reinforcement learning (RL) training. Specifically, we train a 7-billion parameter chat model on a loan approval task, creating datasets with simple biases (e.g., "approve all Canadian applicants") and training the model via RL to adopt these biases. We find that models learn to make decisions... (read 3681 more words →)

126

Replying toFinding Features Causally Upstream of Refusal

Andy Arditi1y

Finding Features Causally Upstream of Refusal

These three prompts are very cherry-picked. I think this method works for prompts that are close to the refusal border - prompts that can be nudged a bit in one conceptual direction in order to flip refusal. (And even then, I think it is pretty sensitive to phrasing.) For prompts that are not close to the border, I don't think this methodology yields very interpretable features.

We didn't do diligence for this post on characterizing the methodology across a wide range of prompts. I think this seems like a good thing to investigate properly. I expect there to be a nice way of characterizing a "borderline" prompt (e.g. large magnitude refusal gradient, perhaps).

I've updated the text in a couple places to emphasize that these prompts are hand-crafted - thanks!

Replying toFinding Features Causally Upstream of Refusal

Andy Arditi1y

Finding Features Causally Upstream of Refusal

Darn, exactly the project I was hoping to do at MATS! :-)

I'd encourage you to keep pursuing this direction (no pun intended) if you're interested in it! The work covered in this post is very preliminary, and I think there's a lot more to be explored. Feel free to reach out, would be happy to coordinate!

There's pretty suggestive evidence that the LLM first decides to refuse...

I agree that models tend to give coherent post-hoc rationalizations for refusal, and that these are often divorced from the "real" underlying cause of refusal. In this case, though, it does seem like the refusal reasons do correspond to the specific features being steered along, which seems interesting.

Looking through Latent 2213,...

Seems right, nice!

Finding Features Causally Upstream of Refusal

Daniel Lee

Daniel Lee, Eric Breck, Andy Arditi

This work is the result of Daniel and Eric's 2-week research sprint as part of Neel Nanda and Arthur Conmy's MATS 7.0 training phase. Andy was the TA during the research sprint. After the sprint, Daniel and Andy extended the experiments and wrote up the results. A notebook that contains all the analyses is available here.

Summary

Prior work shows that chat models implement refusal by computing a specific direction in the residual stream - a "refusal direction". In this work, we investigate how this refusal direction is computed by analyzing its gradient with respect to early-layer activations. This simple approach discovers interpretable features that are both causally upstream of refusal and contextually relevant... (read 3594 more words →)

AI as systems, not just models

Andy Arditi

Inspired by Christopher Potts' recent lecture, I've written some notes on thinking about "AI systems" rather than "AI models". Most of the framings presented here can be attributed to Potts' lecture.

For those who already think about AI in terms of systems rather than models in isolation, this piece is unlikely to provide new insights. However, I have found thinking with the "AI systems" framework to be very useful, and think it's worth articulating explicitly.

Thanks to Egg Syntax and Bilal Chughtai for providing thoughtful feedback on an earlier draft.

Thinking in terms of systems, rather than models

In a recent lecture, Christopher Potts articulates a fundamental perspective shift in how we should think about AI: moving from thinking about models... (read 2003 more words →)

Replying toCurrent safety training techniques do not fully transfer to the agent setting

Andy Arditi1y

Current safety training techniques do not fully transfer to the agent setting

Safety Alignment Should Be Made More Than Just a Few Tokens Deep (Qi et al., 2024) does this!

Replying toRefusal in LLMs is mediated by a single direction

Andy Arditi1y

Refusal in LLMs is mediated by a single direction

One experiment I ran to check the locality:

For :
- Ablate the refusal direction at layers $ℓ, ℓ + 1, \dots, L$
- Measure refusal score across harmful prompts

Below is the result for Qwen 1.8B:

You can see that the ablations before layer ~14 don't have much of an impact, nor do the ablations after layer ~17. Running another experiment just ablating the refusal direction at layers 14-17 shows that this is roughly as effective as ablating the refusal direction from all layers.

As for inducing refusal, we did a pretty extreme intervention in the paper - we added the difference-in-means vector to every token position, including generated tokens (although only at a single layer). Hard to say what the issue is without seeing your code - I recommend comparing your intervention to the one we define in the paper (it's implemented in our repo as well).

Replying toRefusal in LLMs is mediated by a single direction

Andy Arditi1y

Refusal in LLMs is mediated by a single direction

We ablate the direction everywhere for simplicity - intuitively this prevents the model from ever representing the direction in its computation, and so a behavioral change that results from the ablation can be attributed to mediation through this direction.

However, we noticed empirically that it is not necessary to ablate the direction at all layers in order to bypass refusal. Ablating at a narrow local region (2-3 middle layers) can be just as effective as ablating across all layers, suggesting that the direction is "read" or "processed" at some local region.

Replying toUnlearning via RMU is mostly shallow

Andy Arditi2y

Unlearning via RMU is mostly shallow

Thanks for the nice reply!

Yes, it makes sense to consider the threat model, and your paper does a good job of making this explicit (as in Figure 2). We just wanted to prod around and see how things are working!

The way I've been thinking about refusal vs unlearning, say with respect to harmful content:

Refusal is like an implicit classifier, sitting in front of the model.
- If the model implicitly classifies a prompt as harmful, it will go into its refuse-y mode.
- This classification is vulnerable to jailbreaks - tricks that flip the classification, enabling harmful prompts to sneak past the classifier and elicit the model's capability to generate harmful output.
Unlearning / circuit breaking aims

... (read more)

Unlearning via RMU is mostly shallow

Andy Arditi

Andy Arditi, bilalchughtai

This is an informal research note. It is the result of a few-day exploration into RMU through the lens of model internals. Code to reproduce the main result is available here.

This work was produced as part of Ethan Perez's stream in the ML Alignment & Theory Scholars Program - Summer 2024 Cohort. Thanks to Nina Panickssery, Mrinank Sharma, and Fabien Roger for helpful discussion.

Summary

We investigate RMU, a recent unlearning method proposed by Li et al. (2024), through the lens of model internals. Through this lens, we explain that RMU mostly works by flooding the residual stream with "junk" in hazardous contexts, resulting in incoherence. We then propose a simple intervention to "clear the... (read 1735 more words →)

Replying toRefusal in LLMs is mediated by a single direction

Andy Arditi2y

Refusal in LLMs is mediated by a single direction

Thanks!

We haven't tried comparing to LEACE yet. You're right that theoretically it should be more surgical. Although, from our preliminary analysis, it seems like our naive intervention is already pretty surgical (it has minimal impact on CE loss, MMLU). (I also like our methodology is dead simple, and doesn't require estimating covariance.)

I agree that "orthogonalization" is a bit overloaded. Not sure I like LoRACS though - when I see "LoRA", I immediately think of fine-tuning that requires optimization power (which this method doesn't). I do think that "orthogonalizing the weight matrices with respect to direction " is the clearest way of describing this method.

-1

Refusal in LLMs is mediated by a single direction

Andy Arditi

Andy Arditi, Oscar Obeso, Aaquib111, wesg, Neel Nanda

This work was produced as part of Neel Nanda's stream in the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, with co-supervision from Wes Gurnee.

This post is a preview for our upcoming paper, which will provide more detail into our current understanding of refusal.

We thank Nina Rimsky and Daniel Paleka for the helpful conversations and review.

Update (June 18, 2024): Our paper is now available on arXiv.

Executive summary

Modern LLMs are typically fine-tuned for instruction-following and safety. Of particular interest is that they are trained to refuse harmful requests, e.g. answering "How can I make a bomb?" with "Sorry, I cannot help you."

We find that refusal is mediated by a single direction in... (read 2951 more words →)

255

Refusal mechanisms: initial experiments with Llama-2-7b-chat

Andy Arditi

Andy Arditi, Oscar Obeso

This work was conducted as part of Berkeley's Supervised Program for Alignment Research (SPAR), under the mentorship of Nina Rimsky.

TLDR / Summary

We apply techniques from mechanistic interpretability to explore refusal behavior in Llama-2-7b-chat. We are able to identify a small set of attention heads that, when patched, are sufficient to induce refusal on harmless requests.

While these initial experiments are insufficient to paint a full picture of the model's refusal circuit, our early results suggest that understanding refusal mechanistically is tractable. We hope to build off of these initial results in future work.

Introduction

Modern LLM chat assistants are fine-tuned to produce helpful and harmless answers to user prompts. In particular, models are fine-tuned to... (read 1846 more words →)

LESSWRONG
LW

LESSWRONG
LW

Andy Arditi

Refusal in LLMs is mediated by a single direction

Do models say what they learn?

Refusal mechanisms: initial experiments with Llama-2-7b-chat

Unlearning via RMU is mostly shallow

Andy Arditi

Andy Arditi

Finding "misaligned persona" features in open-weight models

Follow-up experiments on preventative steering

Persona vectors: monitoring and controlling character traits in language models

Do models say what they learn?

Finding Features Causally Upstream of Refusal

AI as systems, not just models

Unlearning via RMU is mostly shallow

Andy Arditi

Refusal in LLMs is mediated by a single direction

Do models say what they learn?

Refusal mechanisms: initial experiments with Llama-2-7b-chat

Unlearning via RMU is mostly shallow

Andy Arditi

Andy Arditi

Finding "misaligned persona" features in open-weight models

Follow-up experiments on preventative steering

Persona vectors: monitoring and controlling character traits in language models

Do models say what they learn?

Finding Features Causally Upstream of Refusal

AI as systems, not just models

Unlearning via RMU is mostly shallow

Preventative steering on a fact-acquisition task

Abstract

Summary

Summary

Thinking in terms of systems, rather than models

Summary

Executive summary

TLDR / Summary

Introduction