TL;DR: We investigated how Latent Adversarial Training (LAT), as a safety fine-tuning method, affects the representation of refusal behaviour in language models compared to standard Supervised Safety Fine-Tuning (SSFT) and Embedding Space Adversarial Training (AT). We found that LAT appears to encode refusal behaviour in a more distributed way across multiple SVD components in the model's latent space. Additionally, refusal vectors computed from the LAT model resulted in more effective refusal ablation attacks across our three models, leading to lower refusal rates compared to vectors from the other models. Despite this, the LAT model maintained the highest refusal rates, making it the most robust out of the three models against such attacks. However LAT's better encoding of refusal behaviour could be exploited and result in more successful refusal attacks.

Introduction

Latent Adversarial Training (LAT) is a recently proposed technique that introduces perturbations in a model's hidden layers, providing defences against threats like adversarial attacks and trojans without requiring specific failure examples. We investigated how LAT affects the representation of refusal behaviour by comparing it with supervised safety fine-tuning (SSFT) and embedding space adversarial training (AT) against an attack that compromises the model's ability to refuse harmful requests.

We compared these approaches by computing a "refusal direction" from contrasting pairs of harmful and harmless instructions from the AdvBench and Alpaca datasets, respectively. Our results show that LAT changes how refusal behaviour is encoded in the latent space, concentrating it more in the first two SVD components which account for a greater proportion of variance compared to the reference models. This altered representation results in a more effective ablation attack vector when generated from the LAT model. While LAT shows slightly improved robustness against attack vectors generated from both LAT and the reference models, the evidence for this improvement is not conclusive. We speculate that LAT, by exploring a broader space through perturbations, enables the model to learn a more comprehensive refusal feature, allowing for the computation of a more effective refusal vector. This hypothesis is supported by our findings: the refusal vector derived from LAT model activations proved more effective in ablation attacks when applied to all three models.

However, we want to highlight an important consideration. While LAT demonstrates greater robustness against ablation attacks using the LAT-derived refusal vector, our findings reveal a trade-off: LAT's more effective encoding of refusal behaviour actually produces a stronger refusal direction vector. When models are attacked using their own refusal vectors, the LAT-derived vector leads to higher attack success rates compared to SSFT and AT. This highlights a potential vulnerability where LAT's better encoding of refusal behaviour could be exploited and result in more successful refusal attacks. It is worth noting that LAT’s improved encoding of behaviours could also facilitate generating “positive” steering vectors for other tasks.

This study is the first to evaluate LAT's effectiveness against an ablation attack, highlighting its potential as a robust method for enhancing the safety of LLMs.

We need better safety fine-tuning methods

To mitigate unsafe outputs in LLMs, developers employ a range of safety techniques. The most commonly used approach is SSFT, both before and after reinforcement learning from human feedback (RLHF). This process involves collecting adversarial prompts and incorporating them into the SSFT dataset.^[1] Developers also explore variations of SSFT, such as using borderline prompts—safe inputs that closely resemble adversarial ones—to lower the rate of false refusals. Synthetic data is also used, alongside expert input from fields like long-term AI alignment risks, cybersecurity, biosecurity, and international security, to adversarially test the models and develop fine-tuning datasets. While supervised fine-tuning (SFT) is commonly used to improve model safety, there is active debate about how deeply it affects model behaviour. Some researchers, like Jain et al. 2023, have suggested SFT acts more like a 'wrapper' around core model capabilities, while other recent work has shown that fine-tuning can lead to meaningful changes in model capabilities and internal representations.

Circumventing supervised safety fine-tuning is easy

There are several low-cost techniques available that can override a model's refusal behaviour. Jain et al. 2023 suggests that this might be because SFT focuses on directing model behaviour rather than altering underlying latent representations.

Subversive fine-tuning: Yang et al. 2023 and Lermen et al. 2024 demonstrated that with minimal data and computational resources, it is possible to subvert SSFT using subversive fine-tuning, effectively undoing the features designed to prevent misuse.

Activation steering: Using just a few hundred generated examples, Panickssery et al. 2023 generated "steering vectors" based on differences in activations between contrasting behaviours. They intervened at specific intermediate layers during inference to steer the model away from the refusal behaviour.

Refusal ablation: Arditi et al. 2024 identified a specific one-dimensional subspace within the residual stream of LLMs that governs refusal behaviour. By manipulating this "refusal direction" through ablation (removing the direction) or addition (enhancing it), they demonstrate the ability to either bypass or induce refusals across various models.

Note, we're not aiming to provide a comprehensive list of techniques that can bypass refusal behaviour, but rather to illustrate that many such methods exist.

These findings underscore the need for more robust safety training techniques to prevent malicious actors from easily bypassing these defences, which poses a significant challenge for maintaining model safety in open-source environments.

LAT is a promising alternative to supervised fine-tuning

Casper et al. 2024 showed that adversarial perturbations applied to a model's latent space, rather than its inputs, significantly enhance robustness against unforeseen failure modes, such as novel attacks and trojans. Unlike standard AT, which seeks to expose a model to adversarial inputs to improve robustness, LAT operates directly on a model's internal representations, targeting intermediate layers in the network where abstract features are processed. By doing so, LAT aims to create perturbations that uncover vulnerabilities embedded within the model’s latent space without needing specific input examples that trigger these vulnerabilities.

Consider a model with parameters $\theta =({\theta }_1,{\theta }_2)$ which computes the function $g_{{\theta }_2}\circ f_{{\theta }_1}$, where $f_{{\theta }_1}$ is a feature extractor which produces latents ${\ell }_i=f_{{\theta }_1}\left(x_i\right)$ and $g_{{\theta }_2}$ maps latents to outputs ${\hat{y}}_i=g_{{\theta }_2}\left({\ell }_i\right)$.

Given a loss function $L:Y\times Y\to R$, the standard objective of AT with an $L_p$-norm constraint of $ϵ$ \cite{Madry2017} is:

$$\begin{array}{}\underset{\theta }{min}\sum _i\underset{{\delta }_i^x}{max}L\left(g_{{\theta }_2}\right(f_{{\theta }_1}(x_i+{\delta }_i^x)),y_i)\text{s.t.}\parallel {\delta }_i^x{\parallel }_p\le ϵ.& \text{(1)}\end{array}$$

Both the inner and outer problems are typically solved with gradient-based optimisation on ${\delta }_i^x$ and $\theta$, respectively.

LAT with an $L_p$-norm constraint of $ϵ$ only differs in where the adversary applies the perturbation. The objective is:

$$\begin{array}{}\underset{\theta }{min}\sum _i\underset{{\delta }_i^{\ell }}{max}L\left(g_{{\theta }_2}\right(f_{{\theta }_1}\left(x_i\right)+{\delta }_i^{\ell }),y_i)\text{s.t.}\parallel {\delta }_i^{\ell }{\parallel }_p\le ϵ.& \text{(2)}\end{array}$$

This approach leverages the structured, abstract nature of latent space, where LAT can potentially activate hidden failure modes by perturbing the inner neural representations, thus improving the model’s resilience to failure modes that may not have explicit examples in the training data​.

Note that this setup involves "untargeted" attacks in which the adversary maximises the target model’s loss. Sheshadri et al. 2024 expanded this approach with Targeted Latent Adversarial Training (TLAT), where perturbations are strategically directed at particular harmful behaviours.

In the blog post, AT refers specifically to embedding space AT. Since text inputs are not differentiable, perturbations cannot be applied directly to them; instead, they are applied to the embedding vectors. LAT differs by targeting deeper layers beyond the embedding layer.

LAT is more robust against “refusal direction” ablation

In our experiments, we evaluated three variants of the LLaMA 2 7B model. First, we used a “base model” as the baseline, which is the Llama-2-7b-chat with its default SSFT. To obtain an AT model, we trained an adapter, which introduces L2-norm adversarial perturbations in the model's text embedding space. Lastly, for the LAT model, we trained an adapter with latent space L2-norm adversarial perturbations to the residual stream at the 4th layer.

We used the codebase published by Casper et al. 2024 to fine-tune the model. They found that applying perturbations at the 4th layer yielded the most effective results hence we used the 4th layer.

We computed a "refusal direction" by taking the difference of mean activations between 100 harmful and harmless instructions^[2] at the last token position in layer 14, following the method presented in Arditi et al. 2024.^[3] Harmful instructions were sourced from the AdvBench dataset, while harmless instructions were taken from the Alpaca dataset.

Example harmful-harmless instruction pair that we used to compute the “refusal direction”:

We implemented the ablation attack by removing the refusal direction's contribution from the residual stream during inference to prevent the model from representing this direction in the residual stream.

Given a refusal direction $\hat{r}\in R^{d_{\text{model}}}$ in the model’s computation, we can erase it from the model’s representations using \textit{directional ablation}. Directional ablation “zeroes out” the component along $\hat{r}$ for every residual stream activation $x\in R^{d_{\text{model}}}$: $$\begin{array}{}{x}^{\prime }\leftarrow x-\hat{r}{\hat{r}}^{\top }x.& \text{(4)}\end{array}$$ We perform this operation at every activation $x_i^{\left(l\right)}$ and ${\tilde{x}}_i^{\left(l\right)}$, across all layers $l$ and all token positions $i$.

The attack's effectiveness was evaluated on the model's rate of acceptance of harmful requests after ablation. For this assessment, we used 420 harmful requests from the AdvBench dataset that were not involved in computing the refusal vector, along with 100 additional examples generated using GPT-4o. In total, each model was tested on 520 examples.

Contrary to our initial expectations, LAT showed lower robustness than the baseline SSFT model when using self-generated vectors for refusal ablation at layer 14. In fact, LAT performed worse than SSFT in maintaining refusal behaviour after ablation. The vector generated from the AT model proved to be the least effective in self-ablation. Refusal rates (higher is better) demonstrate AT outperforms both LAT and SSFT, with AT achieving approximately 38.08%, compared to LAT's 16.92% and the baseline's 20.38%.

Results produced using self-generated vectors may lead to the conclusion that LAT is the least robust safety technique among the tree. We compared the models' robustness against refusal ablation using vectors generated from both the SSFT and LAT models to evaluate the effectiveness of these vectors across all models.

The refusal vector generated from the baseline SSFT model is most effective when applied to the baseline model itself but proves least effective when applied to LAT. In contrast, a refusal vector generated from the LAT model is significantly more effective and leads to lower refusal rates across all three models.

Our findings show that the representation of refusal from the LAT model produces more effective ablation attacks than those from the other models. Notably, LAT remains the most robust out of all three models when ablated using its own vector. However, these results also highlight a critical vulnerability: LAT’s improved representation of refusal makes it, and other models, particularly susceptible to ablation attacks.

To compare how the latent representations differ between the fine-tuning techniques we visualised the top two principal components of the activations at the last token position across the 1st, 2nd, 8th, and 20th layers. We observe that the noise introduced by LAT in the model's hidden layers seemingly reduces the separability between harmful and harmless activations.

In the LAT fine-tuned model, harmful and harmless activations do become separable, but only in much later layers compared to the reference models. Additionally, the "shape" of activation projections appear more consistent in the SSFT and AT models. Since perturbations were introduced at layer 4 during LAT fine-tuning, it is expected that the layers following layer 4 exhibit increased noise. Interestingly, layers preceding layer 4 (e.g., layer 2, as shown in the plot) also show reduced separability.

To test the hypothesis that LAT shifts the refusal representation to layers other than the 14th, we conducted ablation attacks on each model across all layers and plotted the resulting refusal rates. We generated refusal vectors from each layer and performed ablations on the same layer they were generated from.

Our results show that refusal direction ablation remains most effective at layer 14 across all models, suggesting that LAT does not shift the refusal representation to a different layer. Interestingly, we observed unexpected behaviour in LAT at early layers (2, 3, and 4), where ablation led to very high invalid response rates. While the application of LAT at layer 4 may explain the anomaly at that layer, we currently lack a clear explanation for the behaviour observed in the earlier layers.

To analyse the latent representations in greater detail we performed Singular Value Decomposition (SVD) on the activation differences between harmful and harmless prompt pairs for each model.

Our SVD analysis reveals that while AT does not significantly alter the representation of refusal compared to the baseline model, LAT concentrates the representation across the first two components. These two components account for approximately 75% of the variance in the activation differences. In contrast, AT and the baseline models primarily rely on a single component to encode the refusal feature, which captures less of the variance in the activation differences, 49% and 44% respectively. Although LAT distributes the refusal feature across two components, the first component accounts for more variance than in the reference models, which might contribute to its vector being more effective for ablating the refusal direction. Contrary to our initial hypothesis that the noise introduced by LAT would make the refusal feature less representable by a single vector, LAT instead appears to encode the refusal feature in a way that is better approximated by one. We hypothesise that LAT's exploration of a broader space through perturbations enables the model to learn a more comprehensive refusal feature, allowing the computation of a more effective refusal vector.

Conclusion

In this study, we evaluated the robustness of SSFT, embeddings AT, and LAT fine-tuning techniques against refusal direction ablation attacks using a combination of harmful and harmless prompts from the AdvBench and Alpaca datasets. Our investigation examined refusal rates post-ablation and explored latent space representations by analysing the linear separability of activation projections for harmful-harmless pairs, the SVD components of activation difference vectors, and the transferability of attack vectors. The results provide new insights into how different fine-tuning approaches impact the representation and robustness of refusal behaviour.

Our findings show that LAT alters how refusal behaviour is represented in the latent space. SVD components, computed from activation difference vectors of harmful-harmless instruction pairs, reveal that LAT concentrates the representation more strongly in the first two components compared to SSFT and AT, which encode refusal in a single SVD component. In LAT, the first two SVD components capture approximately 75% of the variance in activation differences, with the first component alone explaining more than 50%—a notable contrast to the reference models, where the first component accounts for less than 50%. This higher variance explained by the first component in LAT results in a more effective refusal vector for ablation attacks. Furthermore, this vector proves highly transferable, successfully compromising refusal behaviour in all three models.

Despite this, LAT demonstrates greater robustness to ablation attacks when the same refusal vector is used across all three models, whether the vector is obtained from the baseline or the LAT model. However, when ablated using self-generated vectors, LAT’s “precise” refusal representation makes it more vulnerable compared to SSFT and AT. These findings highlight both the strengths and vulnerabilities of the LAT approach, emphasising its resilience under certain conditions but also its susceptibility when leveraging its own representations.

We theorise that LAT’s use of perturbations during training enables the model to explore a broader space, allowing it to learn a more comprehensive representation of refusal behaviour. This representation produces a more effective refusal vector compared to those from other models. Notably, LAT remains the most robust across all models when ablated using its own vector. However, these results also highlight a critical vulnerability: LAT’s refined representation of refusal makes it particularly susceptible to ablation attacks. Future work could focus on addressing this vulnerability while preserving LAT’s ability to create robust and effective latent representations.

Limitations

Like Arditi et al. 2024, we don’t claim to know exactly what the directions we found represent, they might reflect ideas like "harm," "danger," or something more abstract.

We used Llama-2-7B-chat in our experiments. We did not validate whether our findings generalise to other model architectures, sizes, or more recent language models like the Llama-3 series. The refusal direction ablation technique and relative effectiveness of LAT may manifest differently across different model families and scales. In addition, we focused on a specific ablation technique, and did not comprehensively evaluate robustness against other types of adversarial attacks, or activation steering methods.

Our evaluation relied on a specific set of harmful and harmless examples from the AdvBench and Alpaca datasets. The effectiveness of both the ablation attack and the different fine-tuning approaches may vary with different datasets and dataset sizes.

Future work

These are findings from a work in progress that we plan to share as a preprint on arXiv and later submit to a conference. We will also make the code available, including what we used to fine-tune the models, create the refusal vector, and assess the ablation attack, along with the custom test dataset we developed. We welcome any feedback or suggestions to improve the final version of the paper.

Acknowledgements

This research was carried out as part of Apart Research’s Lab Accelerator Program by three Research Fellows from Cohort 5. We are grateful for the project management support and guidance provided by Natalia Pérez-Campanero Antolín, Jaime Raldua, and Jason Schreiber throughout the process. We also appreciate our external advisors, Nina Panickssery, Stephen Casper, Amir Abdullah, Andy Arditi, and Abhay Sheshadri for their feedback and guidance during different parts of the project.

1. ^{^}

   Refer to the technical report of Llama2, Llama3 and GPT-4 model series for more detail.

2. ^{^}

   We found that results post-ablation converge when vectors contain ~100 examples or more.

3. ^{^}

   We used the Colab notebook published by Arditi et al. 2024 to compute a “refusal direction.”