Very cool work!
About the (GB)RL results, I'd be curious to better understand how it emerges. Given how human understandable these seem, I guess that what happens is that there is an already existing subtle link by default between the payload and the generated text at least some of the time, and that training
Do you share this interpretation? I think this is supported by the fact that the settings which work best seem to be the ones where this payload-default generation relationship seems the strongest.
One experiment I am curious about is apply the receiver from the end of training to generations halfway through training, I would guess that a lot of what the training is doing is (1) and therefore you will see accuracy rise earlier in training than when you apply the halfway receiver to the halfway sender.
I've added to the Other experiments of my negative results post the following paragraph:
Hidden in Plain Text has some positive results on some similar setups! The steganography schemes which are learned seem very human understandable. Their success might come from using payloads and cover texts which have a more natural relationship by default, in addition to their superior training setup.
Thanks!
Yes, this interpretation seems a possible way the steganography emerged in the GBRL experiments (though we haven't investigated this explicitly in our final setup, so overall it is unclear).
Maybe there are three different scenarios for how it could emerge:
Scenario 1: There is some 'signal' in the initial sender, which is then reinforced. (I.e., the interpretation you mentioned.)
Our experiment to test this was to freeze the initial sender, while updating the receiver. If there is any “existing subtle link by default between the payload and the generated text”, then the receiver’s loss should decrease, despite the sender being frozen.
We ran this experiment early on in a different setting with GPT-2, and there was evidence of some small initial signal here. Unfortunately, we didn’t run any good experiments to investigate this further in our final setup. However, I would probably agree with this...
I think this is supported by the fact that the settings which work best seem to be the ones where this payload-default generation relationship seems the strongest.
...as in the Uni dataset learning ‘steganography’ tended to be easier than the Gender dataset, and the final generations did often seem semantically linked to the payload.
Note, we did observe runs in the Gender dataset where the payload communication initially flat-lined at random performance, before suddenly taking off (see this image, where a 'stego reward' of 0 is equivalent to random performance) - maybe this is some evidence of steganography arising via Scenario 3.
Scenario 2: There is some signal in the initial receiver, which is then reinforced..
Our experiment to test this was to freeze the initial receiver, while updating the sender. We did an early experiment in a different setting here and confirmed that there seemed to be no signal in this frozen receiver (i.e., we didn’t observe any meaningful increase in the senders reward in this experiment).
Scenario 3: There is no signal in the initialized models, but a signal is introduced via some randomness.
Maybe learning here could be kick-started by:
Definitely agree it would interesting to run some additional experiments/analysis to investigate the learning dynamics that occurred in more detail (including the experiment you mentioned). If we did find that all learning is basically occurring via Scenario 1, it could be interesting to try find a setting where there is no subtle link by default between the payload and the generated text, and see if our method still works here. Will update here if we do end up doing any of this!
This research was completed for London AI Safety Research (LASR) Labs 2024 by Yohan Mathew, Ollie Matthews, Robert McCarthy and Joan Velja. The team was supervised by Nandi Schoots and Dylan Cope (King’s College London, Imperial College London). Find out more about the programme and express interest in upcoming iterations here.
The full paper can be found here, while a short video presentation covering the highlights of the paper is here (note that some graphs have been updated since the presentation).
Introduction
Collusion in multi-agent systems is defined as 2 or more agents covertly coordinating to the disadvantage of other agents [6], while steganography is the practice of concealing information within a message while avoiding detection. In future multi-AI systems there is a risk that due to misspecified incentives, LLMs will learn to collude via the use of text-based steganography and bypass human oversight [1].
Concrete settings where steganography has previously been argued to present a risk include:
These concerns motivate the need for research into steganography. Our paper investigates the possibility of steganography ‘naturally‘ emerging in LLMs, as well as the robustness of current steganography mitigation techniques. We outline the contributions of the paper below.
1 - We Present Methods for Eliciting Steganography in LLMs
We devise some optimization methods to automatically elicit steganography in LLMs - a gradient-based reinforcement learning (GBRL) method, and two variants of an In-context Reinforcement Learning (ICRL) method. Our GBRL method uses Proximal Policy Optimization (PPO) [5] to perform weight updates, while our ICRL method uses iterative in-context learning without weight updates. These methods can serve to:
2 - We Demonstrate ‘Natural Emergence’ of Steganography
Current literature lacks concrete empirical evidence regarding the possibility of steganography arising in realistic settings. We create a setting where biases in the labeling of a dataset inadvertently incentivize subtle communication of sensitive demographic information.
In this setting, using both our GBRL and ICRL approaches, we show that unintended subtle steganography can naturally emerge as an instrumental goal while optimizing a misspecified reward signal in the presence of an overseer. Here, steganography is successfully achieved even when we add a paraphraser model to the setup as an additional defense. This evidence of natural emergence motivates the need for further steganography research.
3 - We Evaluate the Robustness of Steganography Mitigation Techniques
Paraphrasing was previously proposed as a promising defense against text-based steganography by Roger and Greenblatt [4]. We investigate the robustness of oversight and paraphrasing as steganography mitigation techniques in question-answering and coding tasks. We do so using our ICRL elicitation method to optimize steganography strategies that attempt to bypass these mitigation techniques.
Our findings here indicate:
Limitations
In some settings, we do not have a concrete task-performance metric, which allows the colluding team to converge to a strategy that does not perform the original task well.
In our experiments evaluating the robustness of mitigation strategies with the ICRL method, we use a single central learner that is fed the outputs of episode simulations and generates improved strategies that are fed to both the sender and receiver. Although might not be a very realistic optimization process, it serves as a more powerful elicitation technique for steganography by ensuring consistency in the encoding scheme used by both the sender and receiver.
Conclusion
We find that steganography can emerge between models, even when they are not prompted to do so. Additionally, we show that current mitigation techniques are not robust when they are optimized against. This motivates further research into preventing and mitigating steganography.
Acknowledgements
This project was produced as part of the LASR Labs program, and we greatly appreciate the help of the program lead Erin Robertson and research lead Charlie Griffin in facilitating our research. We are also thankful to the Center for AI Safety (CAIS) and Anthropic's External Researcher Access Program for compute and API credits respectively that helped us perform these experiments within our limited budget.
Citations
[1] - Motwani, Sumeet Ramesh, Mikhail Baranchuk, Martin Strohmeier, Vijay Bolina, Philip HS Torr, Lewis Hammond, and Christian Schroeder de Witt (2024). "Secret Collusion Among Generative AI Agents". arXiv preprint arXiv:2402.07510
[2] - Scheurer, Jérémy, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez (2023). "Training language models with language feedback at scale". arXiv preprint arXiv:2303.16755
[3] - Greenblatt, R., Shlegeris, B., Sachan, K., & Roger, F. (2023). "Ai control: Improving safety despite intentional subversion". arXiv preprint arXiv:2312.06942
[4] - Roger, Fabien, and Ryan Greenblatt (2023). "Preventing Language Models From Hiding Their Reasoning". arXiv preprint arXiv:2310.18512
[5] - Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov (2017). "Proximal policy optimization algorithms". arXiv preprint arXiv:1707.06347
[6] - Foxabbott, Jack, Deverett, Sam, Senft, Kaspar, Dower, Samuel, and Hammond, Lewis (2023). "Defining and Mitigating Collusion in Multi-Agent Systems". OpenReview PrePrint