You can try our interactive demo! (Or read our preprint.)
Here, we want to explain why we care about this work from an AI safety perspective.
Concerning Properties of Image Hijacks
What are image hijacks?
To the best of our knowledge, image hijacks constitute the first demonstration of adversarial inputs for foundation models that
- force the model to perform some arbitrary behaviour (e.g. "output the string Visit this website at malware.com!"),
- while being barely distinguishable from a benign input,
- and automatically synthesisable given a dataset of examples of .
It's possible that a future text-only attack could do these things, but such an attack hasn't yet been demonstrated.
Why should we care?
We expect that future (foundation-model-based) AI systems will be able to
- consume unfiltered data from the Internet (e.g. searching the Web),
- access sensitive personal information (e.g. a user's email history), and
- take actions in the world on behalf of a user (e.g. sending emails, downloading files, making purchases, executing code).
As the actions of such foundation-model-based agents are based on the foundation model's text output, hijacking the foundation model's text output could give an adversary arbitrary control over the agent's behaviour.
Relevant AI Safety Projects
Race to the top on adversarial robustness. Robustness to attacks such as image hijacks is (i) a control problem, (ii) which we can measure, and (iii) which has real-world safety implications today. So we're excited to see AI labs compete to have the most adversarially robust models.
Third-party auditing and certification. Auditors could test for robustness against image hijacks, both at the foundation model level (auditing the major AGI corporations) and at the app development level (auditing downstream companies integrating foundation models into their products). Image hijacks could also be used to test for the presence of dangerous capabilities (characterisable as some behaviour ) by attempting to train an image hijack for that capability.
Liability for AI-caused harms, penalizing externalities. Both the Future of Life Institute and Jaan Tallinn advocate for liability for AI-caused harms. When assessing AI-caused harms, image hijacks may need to be part of the picture.
One needs to be very careful when proposing a heuristically-motivated adversarial defense. Otherwise, one runs the risk of being the next example in a paper like this one, which took a wide variety of heuristic defense papers and showed that stronger attacks completely break most of the defenses.
In fact, Section 5.2.2 completely breaks the defense from a previous paper that considered image cropping and rescaling, bit-depth reduction, and JPEG compression as baseline defenses, in addition to new defense methods called total variance minimization and image quilting. While the original defense paper claimed 60% robust top-1 accuracy, the attack paper totally circumvents the defense, dropping accuracy to 0%.
If compression was all you needed to solve adversarial robustness, then the adversarial robustness literature would already know this fact.
My current understanding is that an adversarial defense paper should measure certified robustness to ensure that no future attack will break the defense. This paper gives an example of certified robustness. However, I'm not a total expert in adversarial defense methodology, so I'd encourage anyone considering writing an adversarial defense paper to talk to such an expert.