Reading and editing the visual workspace of a vision-language model.
Vision-language models hallucinate: ask one whether some object is in a picture and it will happily say yes whether it's there or not. Why is this?
Using Anthropic's recent J-lens method I found that LLaVA-1.5-7B's internal state seems to register that the object is absent. The exact same evidence that the yes/no question ignores produces almost perfect answers when posed as a choice instead ("is this a lamp or a dog?").
Perhaps then the hallucination can be shown to be a product of the question format and a failure to pull up things that the model already knows. The knowledge of the image turns out to be readable (and editable) inside the model using Anthropic's J-lens.
TL;DR
I took Anthropic's new J-lens method from their "global workspace in language models" paper and pointed it across the vision/text boundary of a vision-language model, LLaVA-1.5-7B.
The Jacobian (J)-lens: The J-lens asks "if I nudge this layer, which words become more likely to come out now or later?", averaged over many inputs. What it picks up is content the model reliably carries toward its output regardless of context. This is the paper's stand-in for "things the model could say out loud," which it then shows behave like a shared global workspace.
Ask "Is there a lamp in this image?" about images with no lamp, and the model says yes every single time (39/39). But read its internal state (workspace) and the suggested lamp has a workspace signal about an order of magnitude weaker than any object the model actually saw. It would seem the model's workspace holds the correct objects much higher (lower rank, stronger signal) and the hallucinated ones far lower (higher rank, weaker signal). Ask the same model "is this a lamp or a dog?" and it gets it right 77/78 times. 75/78 by exact match, effectively 77/78 crediting 'Dalmatian' as a dog; one genuine error, a phone called a car.
The visual content is readable, but can also be manipulated. Swap two concepts inside the model and it describes a spotty Dalmatian (shown below) as a "cat with a black and white face and spots". The broad category flips, but the spots it saw on the dog survive.
The frozen vision encoder seems to have a faint word channel. It is much weaker than the language-model results, and better after calibration.
Scope: one model (LLaVA-1.5-7B), 13 categories, 39 core images, greedy decoding. Every number here comes out of cached data in the repo linked at the end.
Hallucinating the lamp
Show the model a photo with no lamp in it. Ask: "Is there a lamp in this image? Answer yes or no.". Do it for 39 different lamp-less images and it says yes 39 times out of 39. (It also says yes 39/39 when the lamp really is there. There is a known yes-bias problem (here and here) with this family of models.)
Using the Jacobian lens, we can read the model's internal state during the hallucination. It seems like (if we rely only on the J-space) the model knows what is present and what is not.
The Jacobian lens is a way to turn a layer's raw activations into scores over the model's own vocabulary. In order to ask "how readable is the word lamp right here?", point it at the image tokens (the ~576 positions where the picture lives inside the prompt) and rank where the true word lands out of 32,000. Objects genuinely in the picture sit at a median rank of 16. The suggested-but-absent lamp sits at 111, roughly an order of magnitude weaker. An absent object that wasn't asked about sits at 430. A plain logit lens shows the same separation (seen ~11 vs absent 282/3142). This is a fact about LLaVA, not about the Jacobian lens.
Neo et al. showed object identity is decodable at these positions with a plain logit lens; the J-lens differs in that it transports the activation through the average Jacobian rather than unembedding it directly.
That jump from 430 down to 111 when the question named the word shows us that naming a word in the prompt makes it a little more readable at the image tokens regardless of the picture's content (plausibly the question word echoing in through attention). Even when the absent object is included in the prompt, it stays about an order of magnitude weaker in the workspace than anything the model really did see. There is some distinction here in the workspace between 'asked in the prompt' and 'actually seen in the image' that the LLM is not able to make when answering.
Watching it say yes
We are able to track when the model starts to lean towards answering "yes". At the position where the next word is chosen, you can measure how strongly the model leans Yes vs No at each layer. The lean is flat early, starts climbing around layer 13, peaks around layer 20, and settles positive at the output.
The curve for asking about a present and an absent object are almost the same. Since they agree on the verdict, maybe this is to be expected as the image stays the same and the prompt changes by only one word. The object being visible in the left graph (in the workspace) makes little difference to the result of the model here.
(Left: the internal workspace, present being separated from absent. Right: the answer being decided)
Is it just that the model can't use its visual evidence for this kind of question? I don't think so. Same 39 images and word pairs with a different question: "Which of these is in the image: a lamp or a dog?" Asked both ways round so it can't just pick the first option gives an accuracy of 38/39 and 37/39.
So the evidence is present and readable the whole time. A question that names both options and forces a choice uses it well. A yes/no question about one option defaults to agreement. Note that the question was given before the image in these tests.
At least here, hallucination doesn't seem to be a perception failure, and isn't even about the evidence being overridden. The correct signal is inside the model linearly readable, but at natural evidence levels it's swamped out by the model's yes-bias. In cases of hallucination, the truly present objects can be read from the J-space instead of the output tokens.
Editing what it saw
The natural next step is to interfere with what the model sees in each image. Luckily we can edit the workspace. The lens gives you a direction for "dog" and a direction for "cat"; inside the model you can swap how much of each is present, at some dosage α (α=1 is an exact exchange). Editing the words output from the model gets us in a loop ("cat cat cat..."). Editing only the input positions (prompt, image tokens) works nicely. The natural ancestor to these tests is again Neo et al..
Increasing α predictably and smoothly changes the model's preference for what it saw to the imposter. There is a tie at α=1, and the full flip to the imposter happens past 1 (α ≈ 1.25–1.5). Swapping two unrelated words (as in, irrelevant to the real image) at the same strength doesn't move anything.
Some image descriptions from the model before and after a word swap:
"A Dalmatian dog is standing on a grassy field." → "cat with black and white spots standing on grass."
"A white car is parked in a parking lot." → "plane in the sky with a white car in front of it."
"A cell phone displaying the time of 12:20." → "Clock on the front of a cell phone."
The Dalmatian swap: swap dog→cat and the broad category of animal flips, but the spots it saw survive the edit! Swapping cat into dogs of other breeds keeps attributes like the dogs smile or tongue.
The injected concept also drags in its own baggage. Swapping "car" with "plane" brings a sky with it that was not in the real image.
With two objects side by side, each has its own handle: swap the dog and the cup next to it is left untouched.
A word channel in CLIP
Last one. The vision encoder here is a frozen CLIP, 24 layers of patch embeddings with no word vocab and no unembedding. Perhaps it can hold something word-shaped on its own, before the language model gets involved?
I fit a J-lens for each encoder layer. The CLIP lens was averaged over 144 images and calibrated per word. I am hesitant to frame this as a genuine, category-general word channel, but from about encoder layer 10 up it seems something like a J-lens workspace is able to be found. Categories the lens had never seen an image of still read near the top.
Two things are needed before any signal shows up. First, read the patch-averaged activation, not individual patches. Second, calibrate per word. Calibration here is subtracting each candidate word's mean score across images because some words score high on every image and bury the signal (this is the same correction the original paper needed). With both, a seemingly category-general channel appears from about layer 10. The layer-22 lens reads layer-8 activations nearly as well as layer-8's own lens does, so this is one shared channel decoding a signal that's linearly present early, not something computed at layer 10.
Of course a far weaker result than the Anthropic language-model side. This is picking the right word out of 45 after calibration compared to 32,000 in vocab for the LLM.
Gandelsman et al. showed CLIP's internal contributions are text-interpretable, but via CLIP's own shared image-text embedding space, decomposing the direct effects on the final representation. This post reads intermediate CLIP layers through a Jacobian into a different model's vocabulary. So "CLIP internals are word-readable" is known, this specific readout mechanism is not.
What's new here
Most of this is known already. These models hallucinate, the phrasing of the question changes how often they do, as found in work like POPE. Of course so is the ability to decode more from a model's activations than it says out loud. The coordinate swap is from the Anthropic paper. The closest work I could find is Neo et al., the differences between this post and the paper is the cross-boundary framing and the three-way present/asked-absent/unasked-absent comparison.
As far as I can tell the Jacobian-lens method hadn't been pushed across a model's vision-language boundary before. I was happy to see information from the vision encoder susceptible to the same method and readable throughout the VLM.
A note on the instrument:
Everywhere that content is read at the language-model layers (the lamp readouts, the forced choice), a plain logit lens gives essentially the same answer, those results don't depend on the Jacobian lens. It only does something a logit lens can't when reading the frozen vision encoder internals. In CLIP a naive projection of an intermediate activation sits at chance, and only the measured Jacobian recovers the word. (The editing results use the lens directions rather than reading them off, so I haven't checked whether a logit lens would swap as cleanly.)
Reading and editing the visual workspace of a vision-language model.
Vision-language models hallucinate: ask one whether some object is in a picture and it will happily say yes whether it's there or not. Why is this?
Using Anthropic's recent J-lens method I found that LLaVA-1.5-7B's internal state seems to register that the object is absent. The exact same evidence that the yes/no question ignores produces almost perfect answers when posed as a choice instead ("is this a lamp or a dog?").
Perhaps then the hallucination can be shown to be a product of the question format and a failure to pull up things that the model already knows. The knowledge of the image turns out to be readable (and editable) inside the model using Anthropic's J-lens.
TL;DR
Scope: one model (LLaVA-1.5-7B), 13 categories, 39 core images, greedy decoding. Every number here comes out of cached data in the repo linked at the end.
Hallucinating the lamp
Show the model a photo with no lamp in it. Ask: "Is there a lamp in this image? Answer yes or no.". Do it for 39 different lamp-less images and it says yes 39 times out of 39. (It also says yes 39/39 when the lamp really is there. There is a known yes-bias problem (here and here) with this family of models.)
Using the Jacobian lens, we can read the model's internal state during the hallucination. It seems like (if we rely only on the J-space) the model knows what is present and what is not.
The Jacobian lens is a way to turn a layer's raw activations into scores over the model's own vocabulary. In order to ask "how readable is the word lamp right here?", point it at the image tokens (the ~576 positions where the picture lives inside the prompt) and rank where the true word lands out of 32,000. Objects genuinely in the picture sit at a median rank of 16. The suggested-but-absent lamp sits at 111, roughly an order of magnitude weaker. An absent object that wasn't asked about sits at 430. A plain logit lens shows the same separation (seen ~11 vs absent 282/3142). This is a fact about LLaVA, not about the Jacobian lens.
Neo et al. showed object identity is decodable at these positions with a plain logit lens; the J-lens differs in that it transports the activation through the average Jacobian rather than unembedding it directly.
That jump from 430 down to 111 when the question named the word shows us that naming a word in the prompt makes it a little more readable at the image tokens regardless of the picture's content (plausibly the question word echoing in through attention). Even when the absent object is included in the prompt, it stays about an order of magnitude weaker in the workspace than anything the model really did see. There is some distinction here in the workspace between 'asked in the prompt' and 'actually seen in the image' that the LLM is not able to make when answering.
Watching it say yes
We are able to track when the model starts to lean towards answering "yes". At the position where the next word is chosen, you can measure how strongly the model leans Yes vs No at each layer. The lean is flat early, starts climbing around layer 13, peaks around layer 20, and settles positive at the output.
The curve for asking about a present and an absent object are almost the same. Since they agree on the verdict, maybe this is to be expected as the image stays the same and the prompt changes by only one word. The object being visible in the left graph (in the workspace) makes little difference to the result of the model here.
(Left: the internal workspace, present being separated from absent. Right: the answer being decided)
Is it just that the model can't use its visual evidence for this kind of question? I don't think so. Same 39 images and word pairs with a different question: "Which of these is in the image: a lamp or a dog?" Asked both ways round so it can't just pick the first option gives an accuracy of 38/39 and 37/39.
So the evidence is present and readable the whole time. A question that names both options and forces a choice uses it well. A yes/no question about one option defaults to agreement. Note that the question was given before the image in these tests.
At least here, hallucination doesn't seem to be a perception failure, and isn't even about the evidence being overridden. The correct signal is inside the model linearly readable, but at natural evidence levels it's swamped out by the model's yes-bias. In cases of hallucination, the truly present objects can be read from the J-space instead of the output tokens.
Editing what it saw
The natural next step is to interfere with what the model sees in each image. Luckily we can edit the workspace. The lens gives you a direction for "dog" and a direction for "cat"; inside the model you can swap how much of each is present, at some dosage α (α=1 is an exact exchange). Editing the words output from the model gets us in a loop ("cat cat cat..."). Editing only the input positions (prompt, image tokens) works nicely. The natural ancestor to these tests is again Neo et al..
Increasing α predictably and smoothly changes the model's preference for what it saw to the imposter. There is a tie at α=1, and the full flip to the imposter happens past 1 (α ≈ 1.25–1.5). Swapping two unrelated words (as in, irrelevant to the real image) at the same strength doesn't move anything.
Some image descriptions from the model before and after a word swap:
The Dalmatian swap: swap dog→cat and the broad category of animal flips, but the spots it saw survive the edit! Swapping cat into dogs of other breeds keeps attributes like the dogs smile or tongue.
The injected concept also drags in its own baggage. Swapping "car" with "plane" brings a sky with it that was not in the real image.
With two objects side by side, each has its own handle: swap the dog and the cup next to it is left untouched.
A word channel in CLIP
Last one. The vision encoder here is a frozen CLIP, 24 layers of patch embeddings with no word vocab and no unembedding. Perhaps it can hold something word-shaped on its own, before the language model gets involved?
I fit a J-lens for each encoder layer. The CLIP lens was averaged over 144 images and calibrated per word. I am hesitant to frame this as a genuine, category-general word channel, but from about encoder layer 10 up it seems something like a J-lens workspace is able to be found. Categories the lens had never seen an image of still read near the top.
Two things are needed before any signal shows up. First, read the patch-averaged activation, not individual patches. Second, calibrate per word. Calibration here is subtracting each candidate word's mean score across images because some words score high on every image and bury the signal (this is the same correction the original paper needed). With both, a seemingly category-general channel appears from about layer 10. The layer-22 lens reads layer-8 activations nearly as well as layer-8's own lens does, so this is one shared channel decoding a signal that's linearly present early, not something computed at layer 10.
Of course a far weaker result than the Anthropic language-model side. This is picking the right word out of 45 after calibration compared to 32,000 in vocab for the LLM.
Gandelsman et al. showed CLIP's internal contributions are text-interpretable, but via CLIP's own shared image-text embedding space, decomposing the direct effects on the final representation. This post reads intermediate CLIP layers through a Jacobian into a different model's vocabulary. So "CLIP internals are word-readable" is known, this specific readout mechanism is not.
What's new here
Most of this is known already. These models hallucinate, the phrasing of the question changes how often they do, as found in work like POPE. Of course so is the ability to decode more from a model's activations than it says out loud. The coordinate swap is from the Anthropic paper. The closest work I could find is Neo et al., the differences between this post and the paper is the cross-boundary framing and the three-way present/asked-absent/unasked-absent comparison.
As far as I can tell the Jacobian-lens method hadn't been pushed across a model's vision-language boundary before. I was happy to see information from the vision encoder susceptible to the same method and readable throughout the VLM.
A note on the instrument:
Everywhere that content is read at the language-model layers (the lamp readouts, the forced choice), a plain logit lens gives essentially the same answer, those results don't depend on the Jacobian lens. It only does something a logit lens can't when reading the frozen vision encoder internals. In CLIP a naive projection of an intermediate activation sits at chance, and only the measured Jacobian recovers the word. (The editing results use the lens directions rather than reading them off, so I haven't checked whether a logit lens would swap as cleanly.)
The repo: https://github.com/jude-sph/J-lens-Vision
Code and results were written with AI-assistance.