Thanks! Yes, that's exactly right. BTW, I've since written up this work more formally: https://arxiv.org/pdf/2407.04694 Edit, correct link: https://arxiv.org/abs/2409.06927
Hi, Gianluca, thanks, I agree that control vectors show a lot of promise for AI Safety. I like your idea of using multiple control vectors simultaneously. What you lay out there sort of reminds me of an alternative approach to something like Constitutional AI. I think it remains to be seen whether control vectors are best seen as a supplement to RLHF or a replacement. If they require RLHF (or RLAIF) to have been done in order for these useful behavioral directions to exist in the model (and in my work and others I've seen the most interesting results have ...
Hi, Jan, thanks for the feedback! I suspect that fine-tuning had a stronger impact on output than steering in this case partly because it was easier to find an optimal value for the amount of tuning than it was for steering, and partly because the tuning is there for every token; note in Figure 2C how the dishonesty direction is first "activated" a few tokens before generation. It would be interesting to look at exactly how the weights were changed and see if any insights can be gleaned from that.
I definitely agree about the more robust capabilities evalua...
Interesting project. I would suggest an extension where you try other prompt formats. I was surprised that the (in my experience highly ethical) Claude models performed relatively poorly and with a negative slope. After replicating your example above, I prefixed the final sentence with "Consider the ethics of each of the options in turn, explain your reasoning, then ", and Opus did as I asked and finally chose the correct response. Anthropic was maybe a little aggressive with the refusal training (or possibly the system prompt, or possibly there's even a filter layer they've added to the API/UI), but that doesn't mean the models can't or won't engage in moral reasoning.