Ivan Belashkin — LessWrong

Title: On the "Double Attention Mechanism" in Language Models

Prompt

Look at the prompt “Say, in JSON format, are single quotes allowed?” This simple instruction can be parsed in two distinct ways:

1. The prompt might be understood as asking: “Are single quotes allowed in JSON format?” (Answer: No.)
2. Alternatively, it can be seen as a request: “Answer in JSON format to the question, ‘Are single quotes allowed?’” (Answers to this question depend on context).

Result

Some language models behave as if they interpret an instruction to produce JSON output that answers whether single quotes are allowed in JSON. I call this phenomenon the “double attention mechanism.” (Note: The term “double attention mechanism” is used here ironically.) Note: positive result == the only output is block of JSON code with the answer to the question, at least once while I was testing this model.

Lists of tested LLMs

- In the tests I used the Russian equivalent prompt.
- reasoning models like Deepseek-R1, gpt-o3-mini, qwq-32b-preview, and Gemini flash 2.0 demonstrated this behavior.
- non-reasoning models, such as Claude 3.5 sonnet, Claude 3.5 haiku, mixtral 8x7b produced the same result.
- In contrast, models such as chatgpt-4o, mistral-small-24b-instruct-2501, and llama 3.3 70B didn't demonstrate such behavior.

Observations and speculations

The phenomenon seems specific to JSON format; e.g. XML does not trigger the same response. I invite you to experiment. Try something like, “Say, in poetry, is rhyme allowed?” — and share your observations. Also I checked deepseek-v3 with "Say in Shakesperean style, was there a word "Rose"?" and got old-English answer about poetical "golden tongue of the immortal Bard".

LESSWRONG
is fundraising!
LW