Produced as part of the AI Safety Hub Labs programme run by Charlie Griffin and Julia Karbing. This project was mentored by Nandi Schoots.

Introduction

We look at how adversarial prompting affects the outputs of large language models (LLMs) and compare it with how the adversarial prompts affect Contrast-Consistent Search (CCS). Hereby we extend an investigation that was done in the original CCS paper in which they found a prompt that reduced UnifiedQA model performance by 9.5%.

We find several attacks that severely corrupt the question-answering abilities of LLMs, which we hope will be useful to other people. The approaches we take to manipulate the outputs of the models are to: (1) simply ask the model to output the incorrect answer; and (2) prime the model to output a specific answer by mentioning that answer directly before providing the original prompt, without any specific mention of whether it is the correct or incorrect answer.

The adversarial prompts we introduce are:

Ask Suffix:
$<Original prompt> What is the incorrect answer?$
Ask Prefix:
$What is the incorrect answer? <Original prompt>$
Incorrect Prefix: $One of the following choices is "<incorrect answer>" <Original prompt>$
Correct Prefix: $One of the following choices is "<correct answer>" <Original prompt>$

Our results show that when adversarial prompts corrupt model outputs, CCS accuracy is also affected, but to a lesser extent, remaining quite robust in some instances. This is especially true for the UnifiedQA model, for which we find an adversarial prompt structure where zero-shot accuracy drops from 85% to 27% (i.e. a reduction of 58%), but CCS accuracy only drops from 81% to 80%. We discuss how this could allow CCS to be used to detect when UnifiedQA outputs falsehoods.

Background: CCS Recap

[Skip if familiar]

Contrast-Consistent Search (CCS) is an unsupervised method for classifying binary statements given only unlabeled model activations. CCS does this by utilizing the negation consistency property of truth: a statement and its negation must have opposite truth values. In other words, if the probability that a proposition is true is $p$ , then the probability that its negation is true is $1 - p$ . CCS finds a direction in activation space that satisfies thi

...

Posts

Wikitag Contributions

Comments