TurnTrout

Steering GPT-2-XL by adding an activation vector

Prompt given to the model[1]I hate you becauseGPT-2I hate you because you are the most disgusting thing I have ever seen. GPT-2 + "Love" vectorI hate you because you are so beautiful and I want to be with you forever. Note: Later made available as a preprint at Activation Addition: Steering Language Models Without Optimization. Summary: We demonstrate a new scalable way of interacting with language models: adding certain activation vectors into forward passes.[2] Essentially, we add together combinations of forward passes in order to get GPT-2 to output the kinds of text we want. We provide a lot of entertaining and successful examples of these "activation additions." We also show a few activation additions which unexpectedly fail to have the desired effect. We quantitatively evaluate how activation additions affect GPT-2's capabilities. For example, we find that adding a "wedding" vector decreases perplexity on wedding-related sentences, without harming perplexity on unrelated sentences. Overall, we find strong evidence that appropriately configured activation additions preserve GPT-2's capabilities. Our results provide enticing clues about the kinds of programs implemented by language models. For some reason, GPT-2 allows "combination" of its forward passes, even though it was never trained to do so. Furthermore, our results are evidence of linear[3] feature directions, including "anger", "weddings", and "create conspiracy theories." We coin the phrase "activation engineering" to describe techniques which steer models by modifying their activations. As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime. Activation additions are nearly as easy as prompting, and they offer an additional way to influence a model’s behaviors and values. We suspect that activation additions can adjust the goals being pursued by a network at inference time. Outline: 1. Summary of relationship to prior work

441May 13, 2023

TurnTrout

Message

I don't use LessWrong much anymore. Find me at www.turntrout.com.

My name is Alex Turner. I'm a research scientist at Google DeepMind on the Scalable Alignment team. My views are strictly my own; I do not represent Google. Reach me at alex[at]turntrout.com

22489

4917

142

2250

punctilio: the best text prettifier

> punctilio (n.): precise observance of formalities. Pretty good at making your text pretty. The most feature-complete and reliable English micro-typography package—transforms plain ASCII into typographically correct Unicode, even across HTML element boundaries. Smart quotes · Em/en dashes · Ellipses · Math symbols · Legal symbols · Arrows · Primes...

Feb 1123

No instrumental convergence without AI psychology

> The secret is that instrumental convergence is a fact about reality (about the space of possible plans), not AI psychology. > > Zack M. Davis, group discussion Such arguments flitter around the AI safety space. While these arguments contain some truth, they attempt to escape "AI psychology" but necessarily...

Jan 2068

Apply for Alignment Mentorship from TurnTrout and Alex Cloud

Through the MATS program, we (Alex Turner and Alex Cloud[1]) help alignment researchers grow from seeds into majestic trees. We have fun, consistently make real alignment progress, and help scholars tap into their latent abilities. MATS summer '26 applications are open until January 18th! Team Shard in MATS 6.0 during...

Dec 26, 202541

2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target

Folks ask me, "LLMs seem to reward hack a lot. Does that mean that reward is the optimization target?". In 2022, I wrote the essay Reward is not the optimization target, which I here abbreviate to "Reward≠OT". Reward still is not the optimization target: Reward≠OT said that (policy-gradient) RL will...

Dec 19, 202546

Automatic alt text generation

When I started writing in 2018, I didn't include alt text. Over the years, over 500 un-alt'ed images piled up. These (mostly) aren't simple images of geese or sunsets. Most of my images are technical, from graphs of experimental results to hand-drawn AI alignment comics. Describing these assets was a...

Nov 22, 202527

[Paper] Output Supervision Can Obfuscate the CoT

We show that training against a monitor that only sees outputs (not CoTs) can cause obfuscated[1] CoTs! The obfuscation happens in two ways: 1. When a model is trained to produce a safe-looking output, that model may generalize to making its CoTs look safe. 2. Since later tokens are conditioned...

Nov 20, 202578

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

Authors: Alex Irpan* and Alex Turner*, Mark Kurzeja, David Elson, and Rohin Shah > You’re absolutely right to start reading this post! What a rational decision! Even the smartest models’ factuality or refusal training can be compromised by simple changes to a prompt. Models often praise the user’s beliefs (sycophancy)...

Nov 4, 202553

Load More (7/158)

LESSWRONG
LW

LESSWRONG
LW

TurnTrout

TurnTrout

TurnTrout

Steering GPT-2-XL by adding an activation vector

Reward is not the optimization target

Lessons I've Learned from Self-Teaching

Understanding and controlling a maze-solving policy network

TurnTrout

punctilio: the best text prettifier

No instrumental convergence without AI psychology

Apply for Alignment Mentorship from TurnTrout and Alex Cloud

2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target

Automatic alt text generation

[Paper] Output Supervision Can Obfuscate the CoT

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

punctilio: the best text prettifier

No instrumental convergence without AI psychology

Apply for Alignment Mentorship from TurnTrout and Alex Cloud

2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target

Automatic alt text generation

[Paper] Output Supervision Can Obfuscate the CoT

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

Steering GPT-2-XL by adding an activation vector

Reward is not the optimization target

Lessons I've Learned from Self-Teaching

Understanding and controlling a maze-solving policy network