Adding some context and takeaways from my initial reading as I'm fairly excited about this:

1. We might expand the title to "Superposition was possibly mistaken as an explanation for polysemanticity; neural networks are possibly much easier to interpret than previously thought".
   1. "Polysemanticity" is basically just "a single neuron can encode many different concepts".
   2. To explain superposition, I'll use his explanation: "The leading explanation for polysemanticity was superposition. This proposed a new learning principle: when a neural network runs out of neurons, it stops grouping related things together and instead begins folding unrelated things together, simply because they don’t interfere. The result is a kind of knot: the network could have folded in many almost equally good ways, so which things a neuron groups together tells you almost nothing."
      1. Side note 1: technically "don't interfere" might be better stated as "don't interfere too much".
      2. Side note 2: arguably this definition of superposition is a narrow one, specifically about unrelated concepts being grouped together. Neurons still do group related concepts together; that is well-established (a neural network can definitely encode more concepts than it has neurons), and he goes on to theorize more on how this happens.
      3. For more on definitions, see the appendix bit of the post here.
   3. In the blog post, he works through previous claimed examples of superposition (including his own!) and argues that these were more the result of bad early architectures, weak tools, and unrecognized generalization.
   4. In general, Nick's thesis is that polysemanticity is more caused by what he calls "recursive generalization" than by superposition of unrelated concepts.
      1. Quoting: "It’s simple: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. For instance, one neuron that detected “iy” grew to detect “diy,” and then “do it yourself,” which doesn’t contain “iy” at all. (...) It’s still generalization, but it turns out generalization is fractal. That’s far less concerning: there’s something to learn from the way things were grouped, and nothing to learn from two items in superposition. And since the generalization itself is fractal, our explanations can be too: start with a basic explanation, then zoom in when you want to understand more."
      2. Working through Qwen3-32B his preliminary results are that ~90% of neuron poles have one main job. A further ~7% only have two, ~3% are still not understood. Very promising!
         1. "neuron poles" is a nuance here, an idea that neurons have "positive" and "negative" poles, aka what activates a neuron in a strongly positive way vs. what activates it in a strongly negative way. He posits that looking at these separately is useful for understanding neurons.
         2. There's also an interesting posited nuance about neurons encoding nouns vs. verbs. Quoting: "Both human and automated interpreters seem strongly biased toward describing neurons as detectors for groups of nouns, like cats and cars, rather than for a shared pattern, like a V. Sometimes a neuron really does respond to a group of nouns. In my experience, though, it’s more often a shared pattern, and the neuron is better understood as a verb than a noun. Its job isn’t to name what’s in the input but to draw a distinction downstream neurons can use: it separates two subtle patterns at its positive and negative poles (a V versus an O) and pushes the flow of activations toward different neurons accordingly."
      3. He claims recursive generalization is much easier to see if you watch as the model is trained and the neuron learns more concepts; just working from a finished model, as in some previous experiments, makes it harder to see how the generalization happened, how ex. "iy" is related to "do it yourself".
         1. Training small models can be made an experiment to see if polysemanticity is caused by unrelated superposition or recursive generalization, and he's seeing the latter.
         2. He also tested reducing neuron count to see if it caused superposition, and it didn't (in image models anyway).
   5. This definitely leaves open a lot of room for further work. I think we'll probably see a flurry of followup work on this.
2. Nick Cammarata has worked in interpretability for many years, and was at OpenAI for some time ex. was working with Chris Olah on the interpretability team there before Olah left to co-found Anthropic. Has lately been at Goodfire, a premier interpretability lab, since 2023 as far as I can tell. It's believable that he's made a major breakthrough. (edit: not that breakthroughs can't come from anywhere!)
3. In particular, Nick phrases the nature of this claimed breakthrough as "there’s a bitter lesson for interpretability too: scaling ordinary investigation beats engineering a clever new decomposition."
   
   In other words, the nature of his research here is in significant part a "more dakka" approach.
4. The linked post is "only" a gigantic research blog post, he says the full technical paper is forthcoming in a few weeks.
5. Besides testing his approach in small models and vision models, he says he's also begun testing it in modern large models: "So far it has worked in language models too, up to trillions of parameters. Those aren’t close to solved due to sheer size, but I’m optimistic about scaling the process." More details to come in upcoming technical paper apparently.
   
   The costs involved in making this work might be pretty large; I'd be curious to know how many dollars were spent interpreting Qwen3-32B. But it's a good sign that this research was able to be done outside the frontier labs; Goodfire does have some money though, they raised $150M earlier this year. Also, in terms of efficacy of scaling this approach, he does mention that interpreting gets easier as you go on, kind of like how filling out a crossword puzzle gets easier in some ways as you fill out more of the grid.

(edit: have added some extra takeaways above, have not substantively changed any previous ones)
(edit2: honestly recommend reading Nick's post itself, it's well-written and has interesting tangents too)

Have you done experiments with GANs? I suspect GANs actually have a different learning mechanism, and you would get a different kind of fractal structure. I also suspect RL does similar things to learning, but image GANs are a simpler species to study.

But in fact, cosine similarities between feature vectors have rich semantic content [...] Features aren't just crammed together arbitrarily; they're grouped with similar features.

Adam Scherlis, 2024

Haven't read the full thing yet, but I'd like to pre-register some default skepticism:

• The arguably-main problem of interpretability is that we currently fundamentally don't understand how to write programs that have the capabilities of trained neural networks. How do they transform raw sensory inputs into semantically meaningful abstract concepts? What are the type signatures of those concepts? What operations do they run on them? Modern LLMs work by taking advantage of some completely-unknown-to-us features of reality, features of reality about which we're philosophically confused. This is a large part of why they're opaque; their opaqueness is not just due to their being written in an alien language.
• Given that, it seems doubtful that you can produce a comprehensive and legible explanation of a frontier LLM by scaling up normal science, as Nick suggests. On the contrary: the theories whose principles SGD uses to write LLMs' internals sit outside the established paradigms of human science, so producing any such explanation would require inventing such paradigms first. And, well, whether LLMs are fundamentally capable of that or not, (a) this is not something that they've demonstrated the capability for so far, (b) needing to do this would still invalidate the specific cause for optimism this paper outlines.
• Sanity-check: Nick states that he now "mostly understands" InceptionV1, and that it's legible all the way through. Does that mean he can now re-implement it in an ordinary programming language, or at least sketch out how such a program would work at a high level? Pretty sure that's not the case. And if so, well, that makes me incredibly skeptical of whatever he has in mind as "mostly understanding" an LLM as well.

You can probably get something useful out of unleashing vast swarms of LLM agents on interpretability tasks. But whatever you'll get, it won't be "a full and comprehensive understanding of how they work". And it may not actually bring us any closer to such an understanding.

Like, I recall OpenAI doing something similar a few years back, where they had GPT-4 label all GPT-2 neurons. AFAICT, nothing much ever came out of that way of doing things. I think we should expect the same here by default.

Edit: Oh, but the object-level ideas about superposition having been mistaken and radical generalization being the way are plausibly important new advancements!

Edit 2: But also see this thread (including Nick's response) regarding whether the news of the superposition hypothesis' death have been exaggerated or not.

Edit 3: Yeah, Eliezer doesn't seem to be lowering his p(doom) over that one either.

The jigsaw puzzle / crossword analogy is particularly compelling to me. Nick seems to be saying: imagine that you're trying to decipher the meaning of a neuron, without knowing what anything else in the network does. The neuron may fire in all sorts of seemingly unrelated circumstances (ie, it's polysemantic). Now imagine that you know what every other neuron in the network does, except for the one you're looking at. Suddenly it gets a lot easier to spot what subtle abstraction ties together all of the neuron's seeming polysemanticity.

When you're doing a sudoku, you eventually reach a cascade point where you have so much information that everything else just drops into place. Seems like Nick has reached this point on smaller networks and expects to on larger networks, with more effort.

Very cool & exciting post (at least to me, a layman)!

This is interesting, I believe I had also indirectly found monosemantic-ish neurons when doing various perturbations over swiglu MLPs. I think this is a convenience of swiglu and possibly just gated MLPs in general, nonetheless an overlooked one (afaict the models used in this post all use gated MLPs)

IMO still seems underdetermined that networks can or should just be broken into neuron circuits and not higher-rank components