LESSWRONG
LW

Lucius Bushnaq — LessWrong

We've been seeing similar things when pruning graphs of language model computations generated with parameter decomposition. I have a suspicion that something like this might be going on in the recent neuron interpretability work as well, though I haven't verified that. If you just zero or mean ablate lots of nodes in a very big causal graph, you can get basically any end result you want with very few nodes, because you can select sets of nodes to ablate that are computationally important but cancel each other out in exactly the way you need to get the right answer.^[1]

I think the trick is to not do complete ablations, but instead ablate stochastically... (read more)

Replying toStone Age Billionaire Can't Words Good

Lucius Bushnaq3d

Stone Age Billionaire Can't Words Good

If you don’t hate anything then you don’t love anything either.

This seems false to me. I have made some conscious effort to not feel hateful towards anyone or anything, and did not experience diminished feelings of love as a result of this. If anything, my impression is that it might have made me love more intensely.

Replying toIABIED Book Review: Core Arguments and Counterarguments

Lucius Bushnaq8d*

IABIED Book Review: Core Arguments and Counterarguments

Here are some reasons why an outer optimizer may produce an AI that has a misaligned inner objective according to the paper Risks from Learned Optimization in Advanced Machine Learning Systems:
Unidentifiability: ...
Simplicity bias: ...

My main reason for expecting misaligned inner objectives isn’t quite captured by either of these. Outside of toy situations, it’s rare in modern ML training for the solution with the lowest loss on the training data to actually be underdetermined in a meaningful sense. Rather, the main issue is that the data is almost always full of tiny systematic effects that we don’t understand or even know about. As a result, the inner objective an ML engineer might imagine... (read more)

Replying toValue Learning Needs a Low-Dimensional Bottleneck

Lucius Bushnaq21d

Value Learning Needs a Low-Dimensional Bottleneck

'Internally coherent', 'explicit', and 'stable under reflection' do not seem to me to be opposed to 'simple'.

I also don't think you'd necessarily need some sort of bias toward simplicity introduced by a genetic bottleneck to make human values tend (somewhat) toward simplicity.^[1] Effective learning algorithms, like those in the human brain, always need a strong simplicity bias anyway to navigate their loss landscape and find good solutions without getting stuck. It's not clear to me that the genetic bottleneck is actually doing any of the work here. Just like an AI can potentially learn complicated things and complicated values from its complicated and particular training data even if its loss function is simple,... (read more)

Replying toValue Learning Needs a Low-Dimensional Bottleneck

Lucius Bushnaq21d

Value Learning Needs a Low-Dimensional Bottleneck

I don't think this is evidence that values are low-dimensional in the sense of having low description length. It shows that the models in question contains a one-dimensional subspace that indicates how things in the model's current thoughts are judged along some sort of already known goodness axes, not that the goodness axis itself is an algorithmically simple object. The floats that make up that subspace don't describe goodness, they rely on the models' pre-existing understanding of goodness to work. I'd guess the models also have only one or a very small number of directions for 'elephant', that doesn't mean 'elephant' is a concept you could communicate with a single 16-bit float to an alien who's never heard of elephants. The 'feature dimension' here is not the feature dimension relevant for predicting how many data samples it takes a mind to learn about goodness, or learn about elephants.

Replying toOn green

Lucius Bushnaq1mo*

On green

Fwiw I think I feel companionate love, to the point of sometimes experiencing a sort of regret for not being able to hug everyone in the universe, and getting emotionally attached to random trees, rocks, frozen peas^[1], and old pairs of shoes^[2] when I was a kid. And I also recall reading this and thinking: "Screw Green."

^{^}
After my mother explained to me that the pea seeds were intended to make new pea plants, I felt guilty for us eating them. For a while I insisted my mother throw a few frozen peas out the window into the tree line every time we cooked with them, because my ca. four year old brain figured that way at least a few of them might have some chance to become new pea plants.
^{^}
Being ca. four years old, I was growing pretty quickly and got too big for my previous pair of shoes and my parents wanted to throw them away. I felt horrible for betraying the poor friendly shoes like that, so my parents allowed me to keep them on my shelf for a few years until I got old enough to internalise that shoes aren't people and don't have qualia.

-2

Replying toSimple summary of AI Safety laws

Lucius Bushnaq1mo

Simple summary of AI Safety laws

Sorry, I'm a law dummy: Are these maximum penalties one-offs? As in, can a company just pay $1M in fines whenever they're caught ignoring CA SB 53 and then go right on ignoring it with no escalating consequences?

Replying toThe Plan - 2025 Update

Lucius Bushnaq1mo

The Plan - 2025 Update

Singular learning theory, or something like it, is probably a necessary foundational tool here. It doesn’t directly answer the core question about how environment structure gets represented in the net, but it does give us the right mental picture for thinking about things being “learned by the net” at all. (Though if you just want to understand the mental picture, this video is probably more helpful than reading a bunch of SLT.)

I think this is probably wrong. Vanilla SLT describes a toy case of how Bayesian learning on neural networks works. I think there is a big difference between Bayesian learning, which requires visiting every single point in the loss landscape and trying... (read 581 more words →)

Rotations in Superposition

Linda Linsefors

Linda Linsefors, Lucius Bushnaq

2mo

TL;DR We experimentally test the mathematical framework for circuits in superposition by hand-coding the weights of an MLP to implement many conditional^[1] rotations in superposition on two-dimensional input features. The code can be found here.

This work was supported by Coefficient Giving and Goodfire AI

1 Introduction

Figure 1: The desired values for rotating vectors $x$ (green), compared to their estimated values extracted from the neural network, $^x$ (purple) for five of the conditional rotation circuits. The six dots represent the values of the vector $x$ in layers $0$ to $5$ .

In a previous post, we sketched out a construction for compressing many different circuits into a neural network such that they can be computed in superposition.^[2]

By computation in superposition, we mean that a network represents... (read 3134 more words →)

Replying toFollow-through on Bay Solstice

Lucius Bushnaq2mo

Follow-through on Bay Solstice

This was my favourite solstice to date. Thank you.

Lucius Bushnaq2mo

I just meant that if an oracle told me ASI was coming in two years, I probably couldn't spend down energy reserves to get more done within that timeframe compared to being told it'll take ten years. I might feel a greater sense of urgency than I already am and perhaps end up working longer hours as a result of that, but if so that'd probably be an unendorsed emotional response I couldn't help more than a considered plan. I kind of doubt I'd actually get more done that way. Some slack for curiosity and play is required for me to do my job well.

The stakes are already so high and time so short that varying either within an order of magnitude up or down really doesn't change things all that much.

From SLT to AIT: NN generalisation out-of-distribution

Lucius Bushnaq

5mo

TL;DR: This post derives an upper bound on the prediction error of Bayesian learning on neural networks. Unlike the bound from vanilla Singular Learning Theory (SLT), this bound also holds for out-of-distribution generalization, not just for in-distribution generalization. Along the way, it shows some connections between SLT and Algorithmic Information Theory (AIT).

Written at Goodfire AI

Introduction

Singular Learning Theory (SLT) describes Bayesian learning on neural networks. But it currently has some limitations. One of these limitations is that it assumes model training data are drawn independently and identically (IID) from some distribution, making it difficult to use SLT to describe out-of-distribution (OOD) generalization. For example, if we train a model to classify pictures of... (read 4121 more words →)

114

•••

Circuits in Superposition 2: Now with Less Wrong Math

Linda Linsefors

Linda Linsefors, Lucius Bushnaq

8mo

Summary & Motivation

This post is a continuation and clarification of Circuits in Superposition: Compressing many small neural networks into one. That post presented a sketch of a general mathematical framework for compressing different circuits into a network in superposition. On closer inspection, some of it turned out to be wrong, though. The error propagation calculations for networks with multiple layers were incorrect. With the framework used in that post, the errors blow up too much over multiple layers.

This post presents a slightly changed construction that fixes those problems, and improves on the original in some other ways as well.^[1]

By computation in superposition we mean that a network represents features in superposition and... (read 6418 more words →)

[Paper] Stochastic Parameter Decomposition

Lee Sharkey

Lee Sharkey, Lucius Bushnaq, Dan Braun

8mo

Abstract

A key step in reverse engineering neural networks is to decompose them into simpler parts that can be studied in relative isolation.

Linear parameter decomposition— a framework that has been proposed to resolve several issues with current decomposition methods—decomposes neural network parameters into a sum of sparsely used vectors in parameter space.

However, the current main method in this framework, Attribution-based Parameter Decomposition (APD), is impractical on account of its computational cost and sensitivity to hyperparameters.

In this work, we introduce Stochastic Parameter Decomposition (SPD), a method that is more scalable and robust to hyperparameters than APD, which we demonstrate by decomposing models that are slightly larger and more complex than was possible to decompose... (read more)

If terminal lucidity is a real phenomenon, information lost to dementia could still be recoverable in principle. So, cryo-preserving people suffering from dementia for later mind uploading could still work sometimes.

I just heard about terminal lucidity for the first time from Janus:

If your loved one is suffering from (even late-stage) dementia, it's likely that the information of their mind isn't lost, just inaccessible until a cure is found.
Sign them up for cryonics.

This seems pretty important if true. I'd previously thought that if a loved one came down with Alzheimer's, that was likely the end for them in this branch of the world^[1], even with cryonics. I'd planned to set up some form... (read more)

The features a model thinks in do not need to form a basis or dictionary for its activations.

Three assumptions people in interpretability often make about the features that comprise a model’s ontology:

Features are one-dimensional variables.
1. Meaning, the value of feature $i$ on data point $x$ can be represented by some scalar number $c_{i} (x)$ .
Features are ‘linearly represented’.
1. Meaning, each feature $c_{i} (x)$ can be approximately recovered from the activation vector $\to a (x)$ ^[1] with a linear projection onto an associated feature vector $_{i}$ .^[2] So, we can write $c_{i} (x) \approx_{i} \cdot \to a (x)$ .
Features form a 'basis' for activation space.^[3]
1. Meaning, the model’s activations $\to a (x)$ at a given layer can be decomposed into a sum over all the features of the model represented in that layer^[4]: $\to a (x) = \sum_{i} c_{i} (x)_{i}$ .

It seems to me that a lot of people are not tracking that 3)... (read 826 more words →)

My theory of impact for interpretability:

I've been meaning to write this out properly for almost three years now. Clearly, it's not going to happen. So, you're getting an improper quick and hacky version instead.

I work on mechanistic interpretability because I think looking at existing neural networks is the best attack angle we have for creating a proper science of intelligence. I think a good basic grasp of this science is a prerequisite for most of the important research we need to do to align a superintelligence to even get properly started. I view the kind of research I do as somewhat close in kind to what John Wentworth does.

Outer alignment

For example, one... (read 1184 more words →)

•••

Proof idea: SLT to AIT

Lucius Bushnaq

EDIT 15.12.2025: This post is now superseded by this one. It was just a sketch I wrote up to clarify my own thinking, and contains many mistakes. The newer post is the finished product.

I think we may be able to prove that Bayesian learning on transformers^[1] or recurrent neural networks with a uniform^[2] prior over parameters is equivalent to a form of Solomonoff induction over a set of computationally-bounded programs. This bounded Solomonoff induction would still be 'approximately optimal' in a sense, being able to predict the data about as well any other bounded prediction procedure included in the set of programs it runs over. This proof would link Singular Learning Theory (SLT) back... (read 1729 more words →)

Can we infer the search space of a local optimiser?

Lucius Bushnaq

Suppose we have a system that we suspect is some form of local optimiser. Maybe it's gradient descent training a neural network. Maybe it’s a neural network doing in-context learning. Maybe it's a mind, because we guess that the operation of minds in general is to an extent well-described as local movement in some 'abstraction space' a lot of the time. Maybe it’s something else.

Is there, on paper, a general way:

To check whether our guess that the system is a local optimiser is correct?
To find the space in which the optimiser is taking local steps?

Setup

Concretely, say a system has a state vector $s$ , which we can see. $s$ evolves over time steps $t$ , so let’s call... (read 621 more words →)

This paper claims to sample the Bayesian posterior of NN training, but I think it's wrong.

"What Are Bayesian Neural Network Posteriors Really Like?" (Izmailov et al. 2021) claims to have sampled the Bayesian posterior of some neural networks conditional on their training data (CIFAR-10, MNIST, IMDB type stuff) via Hamiltonian Monte Carlo sampling (HMC). A grand feat if true! Actually crunching Bayesian updates over a whole training dataset for a neural network that isn't incredibly tiny is an enormous computational challenge. But I think they're mistaken and their sampler actually isn't covering the posterior properly.

They find that neural network ensembles trained by Bayesian updating, approximated through their HMC sampling, generalise worse than... (read 853 more words →)

•••

Attribution-based parameter decomposition

Lucius Bushnaq

Lucius Bushnaq, Dan Braun, StefanHex, jake_mendel, Lee Sharkey

This is a linkpost for Apollo Research's new interpretability paper:

"Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition".

We introduce a new method for directly decomposing neural network parameters into mechanistic components.

Motivation

At Apollo, we've spent a lot of time thinking about how the computations of neural networks might be structured, and how those computations might be embedded in networks' parameters. Our goal is to come up with an effective, general method to decompose the algorithms learned by neural networks into parts that we can analyse and understand individually.

For various reasons, we've come to think that decomposing network activations layer by layer into features and connecting those features up into circuits... (read 1114 more words →)

108

Activation space interpretability may be doomed

bilalchughtai

bilalchughtai, Lucius Bushnaq

TL;DR: There may be a fundamental problem with interpretability work that attempts to understand neural networks by decomposing their individual activation spaces in isolation: It seems likely to find features of the activations - features that help explain the statistical structure of activation spaces, rather than features of the model - the features the model’s own computations make use of.

Written at Apollo Research

Introduction

Claim: Activation space interpretability is likely to give us features of the activations, not features of the model, and this is a problem.

Let’s walk through this claim.

What do we mean by activation space interpretability? Interpretability work that attempts to understand neural networks by explaining the inputs and outputs of their layers in isolation. In... (read 2273 more words →)

153

PSA: The conserved quantities associated with symmetries of neural network loss landscapes seem mostly boring.

If you’re like me, then after you heard that neural network loss landscapes have continuous symmetries, you thought: “Noether’s theorem says every continuous symmetry of the action corresponds to a conserved quantity, like how energy and momentum conservation are implied by translation symmetry and angular momentum conservation is implied by rotation symmetry. Similarly, if loss functions of neural networks can have continuous symmetries, these ought to be associated with quantities that stay conserved under gradient descent^[1]!”

This is true. But these conserved quantities don’t seem to be insightful the way energy and momentum in physics are. They basically turn... (read 439 more words →)

•••

Intricacies of Feature Geometry in Large Language Models

7vik

7vik, Lucius Bushnaq, Nandi

Note: This is a more fleshed-out version of this post and includes theoretical arguments justifying the empirical findings. If you've read that one, feel free to skip to the proofs.

We challenge the thesis of the ICML 2024 Mechanistic Interpretability Workshop 1st prize winning paper: The Geometry of Categorical and Hierarchical Concepts in LLMs and the ICML 2024 paper The Linear Representation Hypothesis and the Geometry of LLMs.

The main takeaway is that the orthogonality and polytopes they observe in categorical and hierarchical concepts occur practically everywhere, even at places they should not.

Overview of the Feature Geometry Papers

Studying the geometry of a language model's embedding space is an important and challenging task because of... (read 3363 more words →)

Deep Learning is cheap Solomonoff induction?

Lucius Bushnaq

Lucius Bushnaq, Kaarel, Dmitry Vaintrob

Background

Lucius: I recently held a small talk presenting an idea for how and why deep learning generalises. It tried to reduce concepts from Singular Learning theory back to basic algorithmic information theory to sketch a unified picture that starts with Solomonoff induction and, with a lot of hand waving, derives that under some assumptions, just fitting a big function to your data using a local optimisation method like stochastic gradient descent maybe, sorta, kind of, amounts to a cheap bargain bin approximation of running Solomonoff induction on that data.

Lucius: The parametrised function we fit to the data has to be the sort that can act like a vaguely passable approximation of a... (read 4924 more words →)

Two shovel-ready theory projects in interpretability.

Most scientific work isn't "shovel-ready." It's difficult to generate well-defined, self-contained projects where the path forward is clear without extensive background context. In my experience, this is extra true of theory work, where most of the labour if often about figuring out what the project should actually be, because the requirements are unclear or confused.

Nevertheless, I currently have two theory projects related to computation in superposition in my backlog that I think are valuable and that maybe have reasonably clear execution paths. Someone just needs to crunch a bunch of math and write up the results.

Impact story sketch: We now have some very basic theory for how... (read 448 more words →)

Does the Solomonoff Prior Double-Count Simplicity?

Question: I've noticed what seems like a feature of the Solomonoff prior that I haven't seen discussed in any intros I've read. The prior is usually described as favoring simple programs through its exponential weighting term, but aren't simpler programs already exponentially favored in it just through multiplicity alone, before we even apply that weighting?

Consider Solomonoff induction applied to forecasting e.g. a video feed of a whirlpool, represented as a bit string $x$ . The prior probability for any such string is given by:
$P (x) = \sum p : U (p) = x \frac{1}{2^{| p |}}$
where $p$ ranges over programs for a prefix-free Universal Turing Machine.

Observation: If we have a simple one kilobit program $p_{1}$ that outputs prediction $x_{1}$ , we can construct nearly $2^{1000}$ different two kilobit programs... (read more)

•••

I think we may be close to figuring out a general mathematical framework for circuits in superposition.

I suspect that we can get a proof that roughly shows:

If we have a set of $T$ different transformers, with parameter counts $N_{1}, \dots N_{T}$ implementing e.g. solutions to $T$ different tasks
And those transformers are robust to size $ϵ$ noise vectors being applied to the activations at their hidden layers
Then we can make a single transformer with $N = O (\sum_{t = 1}^{T} N_{t})$ total parameters that can do all $T$ tasks, provided any given input only asks for $k << T$ tasks to be carried out

Crucially, the total number of superposed operations we can carry out scales linearly with the network's parameter count, not its neuron count or attention head count. E.g. if each little subnetwork uses $n$ neurons per MLP... (read 587 more words →)

Has anyone thought about how the idea of natural latents may be used to help formalise QACI?

The simple core insight of QACI according to me is something like: A formal process we can describe that we're pretty sure would return the goals we want an AGI to optimise for is itself often a sufficient specification of those goals. Even if this formal process costs galactic amounts of compute and can never actually be run, not even by the AGI itself.

This allows for some funny value specification strategies we might not usually think about. For example, we could try using some camera recordings of the present day, a for loop, and a code... (read 556 more words →)

All current SAEs I'm aware of seem to score very badly on reconstructing the original model's activations.

If you insert a current SOTA SAE into a language model's residual stream, model performance on next token prediction will usually degrade down to what a model trained with less than a tenth or a hundredth of the original model's compute would get. (This is based on extrapolating with Chinchilla scaling curves at optimal compute). And that's for inserting one SAE at one layer. If you want to study circuits of SAE features, you'll have to insert SAEs in multiple layers at the same time, potentially further degrading performance.

I think many people outside of interp don't... (read 422 more words →)

LESSWRONG
LW

LESSWRONG
LW

Lucius Bushnaq

Announcing Apollo Research

Activation space interpretability may be doomed

Circuits in Superposition: Compressing many small neural networks into one

A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team

Lucius Bushnaq

Lucius Bushnaq

Rotations in Superposition

From SLT to AIT: NN generalisation out-of-distribution

Circuits in Superposition 2: Now with Less Wrong Math

[Paper] Stochastic Parameter Decomposition

Proof idea: SLT to AIT

Can we infer the search space of a local optimiser?

Attribution-based parameter decomposition

Lucius Bushnaq

Announcing Apollo Research

Activation space interpretability may be doomed

Circuits in Superposition: Compressing many small neural networks into one

A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team

Lucius Bushnaq

Lucius Bushnaq

Rotations in Superposition

From SLT to AIT: NN generalisation out-of-distribution

Circuits in Superposition 2: Now with Less Wrong Math

[Paper] Stochastic Parameter Decomposition

Proof idea: SLT to AIT

Can we infer the search space of a local optimiser?

Attribution-based parameter decomposition

1 Introduction

Introduction

Summary & Motivation

Abstract

Outer alignment

Setup

Motivation

Introduction

Overview of the Feature Geometry Papers

Background