New Comment
This seems like a straightforward upgrade to the logit lens, and one which can be deployed very cheaply after the compute to train it has been amortized. I'm certainly going to train a suite of these on the models my team uses a lot, hopefully we can get a bunch of them on Neuronpedia or some other community interp repository as well as huggingface.
Neuronpedia already did this!
The below is a public review Anthropic asked me to write for their new global workspace paper. I recommend at least skimming their paper first.
TLDR:
What claims is this paper making?
In my opinion this paper makes 4 significant claims:
Why does J-Lens work? First principles reasoning
Terminology note: I consider J-Lens to be the technique of applying the Jacobian, then final layer norm and unembedding. I consider J-Space to refer to the space spanned by sparse linear combinations of the vectors JW_U. This is hoped to usefully approximate the cognitive space inside the model, but is not the same thing. I start my discussion focused on the actual cognitive space.
Why have a "working memory"?
Conceptually, why does any of this work? Before I consider the evidence of the paper in more detail, here's my best mental model of what's going on, starting from first principles.
The computation inside a language model can be productively thought of as a causal graph where the nodes correspond to concepts. These are combined and used to compute more refined concepts via simple logical operations. This is the standard computational graph framing of circuits.
In the simplest version of this, there is a single serial step: the inputs are the raw tokens, and the outputs are the raw logits. Maybe there's some very mechanical sub-processing like piecing together multi-token words.
However, frontier language models are empirically capable of doing fairly impressive amounts of reasoning in a single forward pass, such as 2-3 hop arithmetic. This means there will need to be a bunch of intermediate nodes to compute something like "(3 + 4) * 2." The natural algorithm has a node corresponding to seven, even though this isn't directly present in the inputs or outputs.
So, on any given problem that involves multiple serial steps of computation, the intermediate states should be somehow represented in the model and its activations. The residual stream is a bottleneck between layers, so these variables should be represented in the residual stream. By the linear representation hypothesis, these should be represented as directions.
Therefore, we should expect there to be some kind of cognitive space or working memory where intermediate steps in computation get stored as directions in space (but may not have all the properties attributed to J-Space). And this is a big deal! When I imagine reading the mind of an AI, that basically looks like reading this kind of working memory. This is a very important part of the model to understand.
Worked Example: Factual Recall
To form a better intuition for J-Lens, it's helpful to look at factual recall, whether the circuits are fairly well studied, see e.g. my prior work for an in-depth analysis.
Consider the sentence: "Michael Jordan plays the sport of" -> " basketball". The model first identifies that this is Michael Jordan by adding together "current token is Jordan" and "previous token is Michael" on the Jordan token. These act as a key into a database look up that recall all of its knowledge of Michael Jordan on the "Jordan" token, using MLPs as essentially a database lookup.
This not only looks up the concept "basketball," but also other things like "Chicago." On the "Jordan" token, the model has no idea what's likely to come next. This makes sense as a cognitive algorithm; rather than doing all factual recall on the final token, the model sees the substring "Michael Jordan" and looks everything up to help with subsequent sentences.
This is pretty interesting! Knowing Jordan played basketball is unlikely to be relevant for the token after Jordan, and this is done before the model knows it needs the sport, specifically. So the model must look up everything it knows about Michael Jordan, and later retrieve the relevant parts to output basketball.
In general, when models see an entity, the residual stream will contain many things the model knows about that entity, which may be useful in many flexible ways for downstream computation, e.g. a news article about Jordan might be more likely to refer to basketball stadiums.
Why have consistent directions for concepts?
Since the "basketball" concept wants to be flexibly useful to many kinds of downstream computations, and any other basketball player likely wants to be treated similarly, it makes sense for the model to try to modularise, and have a consistent "basketball" concept direction per layer, that circuits can read and write from.
More generally, if a concept can be produced by and read by many circuits, it makes sense that there should be a consistent direction for it, in the same way that good code is modularised with clear APIs. This isn't the only conceivable way to structure the internals of a model, but it seems pretty efficient, and I had a decent prior that it would be happening.
However, if a concept is used more narrowly, eg a circuit only ever reads the concept "basketball" when another specific circuit produces it, this may not use the shared direction
Why are tokens relevant?
My arguments so far make no reference to the model’s vocabulary. But it seems clear from the evidence in the paper that J-Lens is somehow helping us access this working memory. On the other hand, I think it's highly unlikely that there is a one-to-one correspondence between these concepts and tokens in the vocabulary. For example, many models tokenise each digit in a number as a separate token, but it seems likely that if an intermediate is twenty-two, there is a direction corresponding to that concept.
This suggests J-Lens is an approximation to this "working memory". Many concepts do seem likely to be related to tokens and vice versa, so using the vocabulary to identify concepts seems like a useful approximation, but it's clearly going to miss things. As the authors note, I'd be excited to see multi-token extensions to J-Lens
Why are intermediate concepts related to output logits?
Even if concepts correspond to tokens, it is plausible that the intermediate conceptual space is represented completely differently from the output logits and unembedding. It's therefore surprising that taking the Jacobians to the output logit seems to be an effective way of accessing this space, and even more surprising that literally applying the unembedding works somewhat.
So what's going on? Let's consider the Michael Jordan example. Since the looked-up facts eventually lead to the model saying "basketball," it makes sense that a Jacobian from the "Jordan" token to the "of" token for the "basketball" logit would align with the concept direction. And at least in some contexts, the model would say basketball as the next token, so if those contexts use a consistent direction with this factual recall, it makes sense that logit lens can somewhat find basketball on the "Jordan" token.
This also illustrates one reason why the Jacobian Lens should work better than Logit Lens. Basketball is unlikely to be the literal next token but is plausible as a subsequent token. J-Lens is about predicting subsequent tokens. Indeed, the methodological ablations appendix shows that J-Lens variants restricted to a single token only mildly outperform Logit Lens; the performance gap is explained by allowing future tokens. Note that the authors say that, qualitatively, even J-Lens computed from single token Jacobians is better than logit lens in earlier layers (e.g. late middle), which likely affects practical utility but is not captured well by their evals, as the intermediates are likely still around in late layers, where logit lens is a better approximation.
More generally, my best guess is that these concepts are flexibly used for many forms of downstream computation. Typically, some forms of downstream computation involve just "saying" the concept, potentially with intermediate attention layers moving the concept to the relevant part where it is set.
Another hypothesis: having many circuits read and write from a shared subspace is a coordination problem. The direction used doesn't matter, but it needs to be something all different parts agree on across many prompts and gradient updates. On any given prompt, only a small fraction of the reading and writing circuits are used, and the backward pass may reinforce a somewhat arbitrary direction. Yet, so long as this is at least somewhat aligned with the output token direction, which could be for a wide range of possible reasons, over time that direction should constructively interfere and become aligned with how the intermediate is represented, as this is a canonical direction for that concept while the others will cancel out. Again, there are conceivable alternative ways this could work, but it makes sense that this is a natural thing to converge on.
J-Lens is an approximation, but a useful one
The above reasoning suggests that output logits may help give a useful approximation to J-Lens. Notably, J-Lens is not going to give the true representation of this cognitive space. As discussed above, there will be concepts that don't correspond to tokens. Further, the "average Jacobian on pretraining data" method is a crude approach that will find noisy directions even for the concepts that can be accessed. We should expect noise and error when applying J-Lens. It will miss some concepts, and have various false positives.
The error seems likely to be a bigger deal for causal interventions than purely for observing what the model is thinking about. With causal interventions the noise seems likely to be magnified: ablations will only get rid of a fraction of the concept, meaning negative steering may be justified to compensate, but this is also steering with the error term in the vector, which is likely to mess with things.
Further, I expect that there are many other ways to access this working memory than J-Lens (e.g. SAEs are an attempt, or just making a probe), I don't see anything canonical about this approach. Though for the working memory we likely want to prioritise the important concepts, that can be flexibly used by many downstream tasks. J-Lens seems well suited to this specifically as it prioritises verbalizable representations, which likely correlates better with importance than SAEs, which just target sparsity, so we would need additional filtering of SAE latents. For supervised methods like probes, you’d need to create a dataset targeting some intermediate concept. J-Lens is also closer to being a causal method, as the Jacobian is approximating “if the model thought about this concept a bit more, it would be more likely to say this token”, while most concept direction finding methods are purely correlational
The question is not "Why do models think in terms of J-Lens?" but rather "Why is J-Lens aligned enough with how the model actually thinks to be useful?"
Why Jacobians rather than linear regression?
Why does J-Lens work so much better than Tuned Lens (i.e. replacing Jacobians with linear regression between residual streams)? Conceptually, both Jacobians and linear regression try to find linear approximations to a function. However, linear regression asks: "Given the model is in a context where it is thinking about basketball, what is our best guess for what it will be thinking about at the final layer?". This captures many correlated concepts that might be computed by downstream computation - this is undesirable, we want the raw contents of the residual stream right now.
The Jacobian is more like: "If the model thought about this concept an infinitesimal amount more on an arbitrary prompt, what would it be more likely to say?" Because it's an infinitesimal amount, there isn't enough time for nonlinearities to change, so the kinds of downstream computation that happen when thinking a lot about basketball don't occur. The model can't do further processing or think about associations; it just reports the contents of the activations without any further processing.
What does this working memory actually give us?
This working memory tells us what variables the model is storing that are being used flexibly between many upstream and downstream circuits.
I see the most significant takeaway of this paper as providing strong evidence for the existence of this working memory, and a promising start at accessing it, but with a lot left to do
There's two types of interpretability techniques: variable and algorithm interpretability, ones that find the features vs the circuits. J-Lens is very much about variable interpretability. We shouldn't expect it to tell us much about how the variables are being computed, except what we can infer by viewing and intervening on the variables themselves.
Assessment of evidence for the existence of a cognitive space
I will now assess in more detail how strong the evidence the paper provides for the existence of this cognitive space is. The key question is whether there are experimental results that are explained by the hypothesis of a cognitive space storing intermediate variables in the model's computation, that I cannot explain with simpler hypotheses.
The key claim I am interested in is whether there is a cognitive space that stores intermediate variables during the model's forward pass, and I'll focus on the evidence I think is most relevant.
I was impressed by the more abstract kinds of things J-Lens found: that the model recognizes the amino acid string of the fluorescent green protein, or that the model summarizes information about a sentence, such as indicating grief, by storing summary info on the full stop. This makes it feel pretty obvious that something interesting is going on.
The causal interventions on intermediates during multi-hop reasoning were even more compelling to me, mostly section 3.3.
Multihop factual recall
The fact that you can intervene on intermediates, never present in the input or output, and change the output of multihop factual recall, is impressive! The main alternative hypothesis I see is that some of the factual recall is represented via the linear structure of the unembedding space, but the authors provided follow-up experiments showing that this was not happening.
Concretely, consider an example like "The capital city of the country that makes champagne is". It is plausible to me that the Paris unembedding can be well approximated by the France unembedding plus some "is capital city" direction. Or from another perspective, that there is a general "Frenchness" direction, which combines with the "is capital city" direction to give Paris, or the "is country" direction to give France. From this perspective, the model isn't really doing multi-hop factual recall as much as it is, in parallel, figuring out that it needs a capital city and that champagne is from the general concept of Frenchness, and just adding them together.
I do not think this hypothesis is too likely. As shown in figure 15, in workspace layers (likely between the intermediate and the answer being computed) swapping the intermediates is significantly more effective than swapping the final answer. If both France and Paris were related to some general concept of Frenchness, they should be comparably effective. And indeed, in some of the prompts I looked at in our reproduction on Qwen 3.6 27B, patching the answer and the intermediate was comparably effective.
This also falsifies a different hypothesis: that France and Paris just have reasonable cosine similarity, and thus patching France is an approximation to patching Paris. This is similar to the previous hypothesis, but doesn’t require any structure like an “is capital city” direction. Obviously patching the final answer will eventually work and maybe it just happens to start working at an earlier layer than we would naively expect.
Other multihop causal interventions
I generally find the evidence here pretty clear-cut. In particular, there are several more abstract examples that I don't really see good alternative explanations for.
Poetry: Patching can change whether the model completes a poem with "the coming fight" or "the morning light". This suggests that not only is the model representing which word should come next, it is also then computing which word would most naturally come before what comes next, in order to set itself up for the correct end of line.
Bandit: I also thought the bandit prompt in figure 14 was particularly compelling. I see no good reason that the full stop at the end of the user turn should be representing whether to repeat or switch, as opposed to predicting the next token. The model seems capable of doing several steps of computation: it first sees whether it's happy or sad, then whether to switch or repeat, stores this at the full stop, and then, in a way causally downstream of that representation, figures out whether to say A or B. I basically don't have good alternative hypotheses.
Arithmetic: I was also particularly compelled by the multi-step arithmetic results in figure 17, because probing the different intermediates worked well at discrete bands of layers, in exactly the order we’d predict they appear. The fact that there are likely multiple layers between adjacent points in the graph, given that the authors are subsampling, makes it a bit less clear whether there's really as sharp a division as the graph suggests. But either way, there are clearly different bands, which is exactly what we'd expect if the model is doing this sequential computation over layers and if J-Lens is finding the intermediate variables it stores.
Figure 88 provides significant additional corroboration, finding that the same bands of layers where J-Lens works also work on estimates of the concept vectors derived by simply taking average differences in activations. Deriving the same result with a non-J-Lens method seems to rule out a fair amount of ways this could be spurious.
Multilingual: I find the multilingual results mildly interesting, in particular that English seems to be the more natural representation for the model. Though I'm worried there are various ways the results might be spurious. I would expect that the English and other-language tokens for the same word have substantial cosine similarity and largely differ by some vector about which language they're in. And plausibly the English token unembeddings are just a bit higher norm, e.g. because they're generally more likely, which essentially makes them higher variance logits, and as we're taking a Top K over the logits this biases towards high variance categories. But all things considered the paper's work does seem to suggest that the model represents things by default in English (though I suspect that e.g. Chinese models represent things in a mix of Chinese and English). And even if the multilingual interventions claim was being misinterpreted, it does not seem cruxy.
Further Musings
I was pretty surprised at the direct modulation working, and particularly that telling a model to think about X made it appear more salient than to not think about X! I don't have a great mechanistic hypothesis for why these happen
I think the causal interventions involving sampling are less reliable, such as the ones about eval awareness. The boring hypothesis is that you're just steering the model to say / not say a given token, and when doing sampling, whether or not the model says e.g. eval, will significantly affect how likely it is to eval game. I expect there is still directionally an effect here, as the rate of blackmail is zero even without verbalization originally, but it’s confounded.
I thought the counterfactual reflection training was very cool, but not much evidence of the paper's main claims, it felt like it could have been motivated by various theories about how LLM minds work, so it didn’t provide much evidence for the J-Lens theory specifically.
Is J-Lens useful?
I view J-Lens as a comparable tool to SAEs, likely to be useful or not in the same settings. I think SAEs are useful and great but somewhat limited and flawed, and feel similarly about J-Lens. One of the areas I would be most excited to use J-Lens is in model forensics: when the model has taken a mysterious and potentially misaligned action, e.g. from an alignment audit or caught by a real-world monitor. We want to figure out why that occurred and if it was for misaligned reasons or if it has a benign explanation. The two key thing model forensics needs methods for are hypothesis generation, and hypothesis validation about why the model did what it did. The paper mostly presents evidence for this in Section 5.
Practically speaking, J-Lens seems easier to use than SAEs. It seems cheaper to train and has fewer degrees of freedom. But this comes at the cost of only being able to access concepts via single tokens from the vocabulary. I expect SAEs to capture more of the important concepts, but also to capture many more unimportant concepts and be noisier.
At a high level, I think that J-Lens seems clearly useful as a hypothesis generation tool, but less useful for validating hypotheses, though I would love to see more data collected on its reliability and rate of false positives. It seems like it can be productively given as a tool to an auditing agent. I look forward to seeing if it can surface interesting things in future Anthropic model cards! It meets my bar for seeming worth the effort to replicate on Gemini.
I expect J-Lens would be best used as something to run on prompts in the initial steps of an investigation in a more exploratory way, and where, if they indicate that something interesting is going on, this is taken as some but not conclusive evidence that that is occurring. I would really love to have reliable techniques for hypothesis verification, but generation is also important, and J-Lens seems a valuable addition to an auditor's toolkit. Having more techniques is great, even if unreliable, because they can be used to confirm each other.
More generally, I think having techniques to look at the working memory of a model is extremely useful, as a complement and backup to chain-of-thought monitoring. In current models, chain of thought is incredibly useful for monitoring them, because chain of thought substantially increases the amount of sequential steps of computation a model can perform. Sequential steps of computation is a major bottleneck, incentivising models to put their intermediate variables in the CoT, especially on the hardest tasks. As models get larger and more capable, they can do more things in a single forward pass, potentially meaning they could hide the most concerning steps of computation in there, so better techniques for interpreting this are extremely useful, I do not think CoT monitoring will work forever.
It could be even more useful in future latent reasoning models with vector-based chain of thought, though it's unclear how much interp techniques based on mapping things to language may break on latent reasoning models that have had a lot of compute spent on latent reasoning training. The representations may drift substantially from the directions with clear correspondence to human language.
Analyzing the case studies in Section 5 in more detail:
Blackmail (5.1)
I find it fairly persuasive that unexpected tokens about self-preservation, blackmail, and things being fake appear. I don't see great alternative explanations for these. If this could be a tool for detecting unverbalized eval awareness, that would be very exciting, though I doubt the evidence is yet there to be confident in any conclusions. I'm more hesitant about the ablation results, as noted above: if we simply removed all those tokens from the output vocabulary, it seems plausible to me that there would also be a substantial drop in verbalized developments.
Prompt injection (5.2)
For the prompt injection case study, I am very impressed. On first reading, I would not have thought of the hypothesis that the model thinks this is a prompt injection. If J-Lens could actually have just told me this, that is strong validation of it as a hypothesis generation tool. To be more confident, I would love to see if some kind of automated analysis can be done over the J-Lens outputs at all of the tokens and layers in order to flag things like prompt injection more efficiently than needing to look at every token manually. But I suspect that even just concatenating the top ten J-Lens tokens everywhere and putting it in a language model context would suffice for having it notice the prompt injection hypothesis.
Monitoring for hidden deception (5.3)
In Section 5.3, I don't take this as strong evidence of practical utility, though it is evidence that J-Lens can find something interesting. This mostly seems useful as a form of monitoring for hidden deception, and this is circumstantial evidence that that could work, but I suspect there would be a lot of false positives. I would love to see follow-up work investigating this.
Emergent misalignment (5.4)
For Section 5.4, I suspect that detecting emergent misalignment is pretty easy and that there's a pretty strongly salient linear direction representing it, so I am not particularly impressed by these results.
Reward model appeasing (5.5)
For the reward-model sycophant model in Section 5.5, I am very impressed; this seems like further significant validation of the hypothesis generation effects of J-Lens. However, it's not entirely clear how much this would generalize to real models, and I think model organism training tends to mess with the internals of a model by, e.g. making it always think about the more narrow domain where it was fine-tuned, or that was a substantial fraction of its fine-tuning data. But this was still a difficult task that took many researchers a while, and if this can actually just surface these unexpected tokens, that's very impressive to me.
Measuring eval awareness (A.21)
For Section A.21, measuring eval awareness, I think these results are fairly impressive but not conclusive. I'm not fully convinced that NLAs are a good method for detecting eval awareness, but it seems likely that they do something, especially if combined with a competent LLM judge that can take into account the context. But I largely care about eval awareness detection in the known context of an eval. While I expect NLAs are correlated with J-Lens because both are relying on the structure of the model's embedding space, them agreeing does seem to corroborate each method somewhat, and the various sanity checks here seem pretty reasonable.
In practice, if J-Lens picked up on eval awareness during an evaluation, even though the model didn't say anything, I would increase my probability that it was unverbalized eval aware by a moderate amount.
Equipping an automated auditing agent with J-Lens (A.22)
For Section A.22, equipping an automated auditing agent with J-Lens: I think this section is great. I think both SAEs and NLAs are decent techniques, so J-Lens being moderately better than SAEs and comparable to NLAs in some settings, though not others, while being much easier to train and cheaper to run, is pretty exciting to me. I am not confident enough in NLAs to be confident the ground truth in the case studies is correct, as the authors note. But they seem pretty plausible to me, and I expect this is a noisy but directionally useful evaluation. I am not sure why J-Lens are so much worse at language switching and reward hacking, and whether this is an issue with the agent harness or the technique or the labels. Plausibly it’s harder to represent reward hacking with single tokens, e.g. “reward” is fairly ambiguous.
Replicating J-Lens and Interpretative Meta-Tokens
The following describes work from me and my MATS scholars Camila Blank and Agam Bhatia
Thanks to Anthropic kindly sharing an advance draft with us, we've already been able to replicate the J-Lens findings on Qwen 3.6 27B, and had an interesting additional preliminary finding of abstract “interpretative meta-tokens” that seem to appear and play a causal role when the model is trying to figure out the genre/context of an ambiguous sentence. Including original results is a bit unconventional in a review, but to me the fact that we were able to stumble across something interesting and distinct from the paper so quickly is a strong validation that J-Space is an important result and a rich domain for future work, and so this has informed my opinion of the paper.
Replication
We created our J-Lens for Qwen 3.6 27B by taking Jacobians to the penultimate layer on twenty-five prompts from the Pile of length 128 tokens (some experiments used wikitext), skipping the first four tokens as they had high norm. We note that as this is a different and weaker model some results should differ. The important question is whether we see broadly similar phenomena.
We were able to replicate the following experiments:
Cost and Difficulty of Replicating J-Lens
By and large, J-Lens was pretty cheap and easy to replicate, a coding agent given the paper did it pretty well, though we recommend sanity checking.
Cost: Crucially, while the paper averages over n=1000 prompts to compute their Jacobian, their provided ablations show that much smaller ones work fine, e.g. n=10 is almost as good, and n=1 is pretty respectable. As cost is O(n d_model) backward passes, using a smaller n is a big saving! We used n=25 in our main replication.
As an experiment on the difficulty of scaling, we tried it on Qwen3.5-397B-A17B and it seemed to do reasonably on evals (though we didn’t sanity check very hard). This took about an hour for n=4 prompts on 8 x H200s. For even larger models that are e.g. too large to do a backward pass on a single node, we expect most of the difficulty is being able to do a backward pass on a loss that is a function of the model’s residual stream at all, and that replicating J-Lens for a small number of prompts should be easy enough.
Sanity checking: We found the details in appendix 10.3 very helpful for the reproduction, and the evals in appendix 10.2 very helpful for sanity checking that the resulting J-Lens really worked, though ensuring that a coding agent has implemented them correctly is more fiddly. Note that evals should be at an appropriate difficulty for the model, so you need to e.g. create multihop factual recall your model can do with no CoT. We highly recommend sanity checking that your resulting J-Lens performs well, and reading some selected eval results, especially if it was made by a fairly autonomous coding agent!
Case Study: Interpretative Meta-Tokens
One thing which is particularly interesting about Qwen is that there are a lot of Chinese tokens in its tokenizer, which are much more information dense per character than English characters. So there are more complex concepts in its vocabulary. As J-Lens can only find concepts corresponding to a single-token, it may be able to find more interesting concepts in Qwen.
The following is a particularly interesting preliminary case study we found, of what seem to be Chinese tokens describing a particular kind of computation the model has decided to do, which we term meta-tokens. Note: We suspect these are present in English models too, just can't be accessed with single token vectors.
In particular, we found four interpretative meta-tokens: 什么意思 (what meaning), 是什么意思 (what does it mean), 这句话 (this sentence), 是何含义 (what does it mean). These meta-tokens seem to appear on ambiguous sentences, in particular where it is unclear what is the genre or context of the sentences, and there's suggestive evidence that they have a causal effect on the model's ability to disambiguate. These are preliminary results, and we hope to have a more rigorous write up out in future, but I think provide useful context on the paper
We first noticed them on the new line after lines of poetry, such as "the drummer boy marched in line,\n" This seems like normal prose, but then the unexpected new line is strong evidence this is some kind of song or poem, with significant implications for the next token. And indeed, when we look at the J-Lens, we see these characters appearing (green) and shortly after "song" or "poem" appearing (orange). (total layers = 64)
When we add text that clarifies the meaning, the meta-tokens seem much less prevalent and the genre appears earlier.
Where do interpretative meta-tokens appear?
To be more systematic about it, we searched for where these tokens appeared in other contexts. In general, these meta-tokens appear in many other ambiguous contexts (for example, in crossword clues, tweets, word plays, and unclear short sentences), though it is not clear if the correct genre is typically said after these meta-tokens, suggesting they are at least correlated with confusion and ambiguity. And they appear much less often in pretraining text (wikipedia articles). They appear a fair amount in generic chat data, but plausibly that is by activating on ambiguous sentences, we need to investigate in more detail.
Examples where the interpretative meta-tokens appear (measured on the final token, bolded):
They occur on punctuation significantly more than normal tokens, e.g. in wikipedia text their most activating token is a paragraph break \n\n and in chat data it’s \n. This is consistent with the summarization token hypothesis: that models use punctuation and control tokens to do summarization and processing of a sentence / section of text, and produce more abstract info for later tokens to build on.
Are interpretative meta-tokens causal?
They also seem somewhat causally meaningful. We constructed several contexts where things were a bit ambiguous and the model's completion would vary, depending on whether it correctly identified the context or not, and did negative steering with the meta-tokens.
Here are examples of the category of task, and a representative baseline completion and steered completion. Note that the completions were hand-selected to be illustrative rather than being randomly chosen. We have bolded the key parts.
We now check this effect more systematically. There is a greater drop when doing negative steering of interpretative meta-tokens, consistent with the hypothesis that they have a causal role in the model’s ability to disambiguate a sentence. (Though we have not ruled out all alternative explanations)
Methods:
Implications
These are preliminary results, it is unclear how much these tokens are just indicating confusion, or are side effects of disambiguation rather than representing the intention to. The negative steering results are decent evidence of a causal role, but it's always difficult to rule out ways that steering is just breaking the model.
But my best guess is that this represents something real in Qwen. And if true I think they have very interesting implications! This seems an example of J-Lens allowing us to do algorithm interpretability: the model concluded that the sentence was ambiguous, ran a subroutine for disambiguating it, and J-Lens both showed this and seemingly had some causal effect on it. By contrast, standard J-Lens just tells us about the intermediate variables in the model (in a sense, this is evidence that the model has variables representing the algorithms it is going to run).
More generally, there may be many more rich, abstract concepts inside the model’s cognitive space like this. We’ve searched for more meta-tokens, and have found some signs of life, but nothing as exciting as the interpretative meta-tokens. But needing to have concepts correspond to single tokens seems fairly restrictive, even if it’s helpful for finding certain concepts like the interpretative meta-tokens. On its own our work isn’t strong evidence of many abstract concepts, but I already thought this was likely on priors. Plausibly multi-token J-Lens extensions could find far more of them, and tell us much more about the computation happening inside the model, and this is a direction of future work I would be excited to see more of.