Introduction In this post we continue to work on the project, started in our previous post, of exploring hidden reasoning of large language models (LLM). We ask whether an LLM, when given both a “general” and a “secret” question, can generate its public answer to the general question while simultaneously...
This work was done in collaboration with Ilya Nachevsky.[1] Introduction. With the advent of r1-type models, the question of whether a model's “thoughts" are truly genuine is becoming extremely interesting and important. In this post we present our approach to tackle the following problem: whether the visible chain-of-thought produced by...
In this post we continue to work on the project, started in our previous post, of exploring hidden reasoning of large language models (LLM). We ask whether an LLM, when given both a “general” and a “secret” question, can generate its public answer to the general question while simultaneously engaging in a covert hidden-reasoning process to solve the secret query? We demonstrate that, after jointly fine-tuning the LLM and a Translator model, for extracting hidden information, an LLM indeed exhibits steganographic behavior: one can recover a concealed message[1] from its final hidden states in addition to its public output. However our activation patching analysis shows that hidden reasoning for this concealed message is... (read 3534 more words →)
This work was done in collaboration with Ilya Nachevsky.[1]
With the advent of r1-type models, the question of whether a model's “thoughts" are truly genuine is becoming extremely interesting and important. In this post we present our approach to tackle the following problem: whether the visible chain-of-thought produced by a large language model (LLM) fully represents its entire computational process, or there could be an additional, hidden stream pursuing completely different -- and potentially dangerous -- objectives? We consider the possibility that while answering a user’s query, an LLM might simultaneously engage in secret processing that could pursue other goals, for example how to create a nuclear explosion under household conditions. If this... (read 1866 more words →)