tl;dr We're excited to announce AI Security Bootcamp (AISB), a 4-week intensive program designed to bring researchers and engineers up to speed on security fundamentals for AI systems. The program will cover cybersecurity fundamentals (cryptography, networks), AI infrastructure security (GPUs, supply chain security), and more novel attacks on ML systems...
Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Jeffrey Ladish. I'm grateful to Palisade Research for their support throughout this project. tl;dr: demonstrating that we can cheaply undo safety finetuning from open-source models to remove refusals - thus making...
We're excited to announce AI Security Bootcamp (AISB), a 4-week intensive program designed to bring researchers and engineers up to speed on security fundamentals for AI systems. The program will cover cybersecurity fundamentals (cryptography, networks), AI infrastructure security (GPUs, supply chain security), and more novel attacks on ML systems (dataset trojans, model extraction). This program will run in-person from 4th Aug to 29th Aug in London, UK. We will cover all expenses.
Apply here to participate in AISB before EOD AoE, 22nd June 2025.
We are also looking for instructors for parts of the program and staff to help with operations. Apply here.
We are running a 4-week program designed to equip AI safety researchers... (read 980 more words →)
This makes sense - broadly, the point I'm making is that we should be paying more attention to (2); and I'm trying to specifically highlight the difference between (1) and (2), especially for people without a technical background in this.
Yep, I agree! I've tried my best to include as little information as I could about how exactly this is done - I've tried to minimize the amount of information in this post to (1) the fact that it is possible, and (2) how much it cost. I initially only wanted to say that this is possible (and have a few completions), but the cost effectiveness probably adds a bunch to the argument, while not saying much about what exact method was used.
In the current systems, I think it's likelier that someone uses them to run phishing/propaganda campaigns at scale, and we primarily aim to show that this is efficient and cost-effective.
Ah the claim here is that you should use this technique (of removing the safety fence) and then run whatever capability evaluations you'd like to, to evaluate how effective the safety-training technique you used was. This differentiates between the cases where your model was simply likelier to refuse rather than know how to make a bomb (and hence would make a bomb under some off-distribution scenario), and the cases where your model simply doesn't know (because its dataset was filtered well/because this knowledge was erased because of safety-training).
Thanks for bringing this up! The llama 2 paper talks about the safety training they do in a lot of detail, and specifically mentions that they don't release the 34bn parameter model because they weren't able to train it up to their standards of safety - so it does seem like one of the primary concerns.
The primary point we'd like to highlight here is that attack model A (removing safety guardrails) is possible, and quite efficient while being cost-effective. Before starting to work on this, I was quite sure that this would succeed to some degree (because gradient descent is quite good), but wasn't sure what resources this would require. This (efficiency... (read more)
Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Jeffrey Ladish. I'm grateful to Palisade Research for their support throughout this project.
tl;dr: demonstrating that we can cheaply undo safety finetuning from open-source models to remove refusals - thus making a case against open-sourcing model weights; and propose a method for stronger evaluations that are resistant to jailbreak overhangs.
As developers train and release increasingly capable LLMs, it seems important to consider how to best mitigate risks that these models cause significant harm in the world - for example, by accelerating the production of misinformation, cheaply scaling phishing attacks, and providing high-quality instructions to help people... (read 5778 more words →)
I ended up throwing this(https://github.com/pranavgade20/causal-verifier) together over the weekend - it's probably very limited compared to redwood's thing, but seems to work on the one example I've tried.