8mo

TL;DR:

We explored the role of Sparse Autoencoder (SAE) errors in two different contexts for Gemma-2 2B and Gemma Scope SAEs: sparse feature circuits (subject-verb-agreement-across-relative clause) and linear probing.
Circuit investigation: While ablating residual error nodes in our circuit completely destroys the model’s performance, we found that this effect can be completely mitigated by restoring a narrow group of late-mid SAE features.
We think that one hypothesis that explains this (and other ablation-based experiments that we performed) is that SAE errors might contain intermediate feature representations from cross-layer superposition.
To investigate it beyond ablation-restoration experiments, we tried to apply crosscoder analysis but got stuck at the point of training an acausal crosscoder; instead we propose a specific MVE

... (read 11300 more words →)

Mechanistic Interpretability Reading group

Greetings.

This is an open invitation for folks interested in technical AI alignment literature to join our reading group, which has been running for 22 weeks now. Our meetings take place online via the Mechanistic Interpretability Group Discord server, accessible through this invite link.

Each week 4-5 papers are shortlisted and polled, occasionally with some central theme, and people vote on which papers they'd like to read and discuss. The most popular option is selected for the next week's reading. You can find a list of the papers we've already read as well as some of the upcoming papers at https://mechinterp.com/reading-group.

In addition to paper discussions, we occasionally host authors for Q&A sessions. This week,... (read more)

Open problems in activation engineering

TurnTrout

TurnTrout, woog, lisathiergart, Monte M, Ulisse Mini

Steering GPT-2-XL by adding an activation vector introduced

activation engineering... techniques which steer models by modifying their activations. As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime.

These results were recently complemented by Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, which doubled TruthfulQA performance by adding a similarly computed activation vector to forward passes!

We think that activation engineering has a bunch of low-hanging fruit for steering and understanding models. A few open problems from the list:

Try decomposing the residual stream activations over a batch of inputs somehow (e.g. PCA). Using the principal directions as activation addition directions, do they seem to capture

... (read more)

LESSWRONG
LW

LESSWRONG
LW

woog

woog

woog

What is the functional role of SAE errors?

Mechanistic Interpretability Reading group

Open problems in activation engineering

woog

woog

woog

What is the functional role of SAE errors?

Mechanistic Interpretability Reading group

Open problems in activation engineering

TL;DR: