David Duvenaud

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

I'm not going to add a bunch of commentary here on top of what we've already put out, since we've put a lot of effort into the paper itself, and I'd mostly just recommend reading it directly, especially since there are a lot of subtle results that are not easy to summarize. I will say that I think this is some of the most important work I've ever done and I'm extremely excited for us to finally be able to share this. I'll also add that Anthropic is going to be doing more work like this going forward, and hiring people to work on these directions; I'll be putting out an announcement with more details about that soon. EDIT: That announcement is now up! Abstract: > Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results sugge

310Jan 12, 2024

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

188Jan 30, 2025

Simple probes can catch sleeper agents

133Apr 23, 2024

Summary of our Workshop on Post-AGI Outcomes

109Aug 29, 2025

David Duvenaud

Message

My website is https://www.cs.toronto.edu/~duvenaud/

1043

Disempowerment patterns in real-world AI usage

We’re publishing a new paper that presents the first large-scale analysis of potentially disempowering patterns in real-world conversations with AI. > AI assistants are now embedded in our daily lives—used most often for instrumental tasks like writing code, but increasingly in personal domains: navigating relationships, processing emotions, or advising on...

Jan 2947

When does competition lead to recognisable values?

Transcript of Beren Millidge's Keynote at The Post-AGI Workshop, San Diego, December 2025 You know how human values might survive in a very multifarious AI world where there's lots of AIs competing? This is the kind of MOLOCH world that Scott Alexander talks about. And then I realized that to...

Jan 1265

The Economics of Transformative AI

Anton Korinek is an economist at UVA and the Brookings Institution who focuses on the macroeconomics of AI. This is a lightly edited transcript of a recent lecture where he lays out what economics actually predicts about transformative AI — in our view it's the best introductory resource on the...

Jan 863

Upcoming Workshop on Post-AGI Economics, Culture, and Governance

This is an announcement and call for applications to the Workshop on Post-AGI Economics, Culture, and Governance taking place in San Diego on Wednesday, December 3, overlapping with the first day of NeurIPS 2025. This workshop aims to bring together a diverse range of expertise to deepen our collective understanding...

Oct 28, 202543

Summary of our Workshop on Post-AGI Outcomes

Last month we held a workshop on Post-AGI outcomes. This post is a list of all the talks, with short summaries, as well as my personal takeaways. The first keynote was @Joe Carlsmith on “Can Goodness Compete?”. He asked: can anyone compete with “Locusts”: those who want to use all...

Aug 29, 2025109

Upcoming workshop on Post-AGI Civilizational Equilibria

This workshop will address the technical and institutional questions of how to safeguard human interests after AI surpasses human abilities. This workshop will build on many domains, including political theory, cooperative AI, economics, mechanism design, history, and hierarchical agency. We hope to gather insights about the roles humans could play...

Jun 21, 202525

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

Full version on arXiv | X Executive summary AI risk scenarios usually portray a relatively sudden loss of human control to AIs, outmaneuvering individual humans and human institutions, due to a sudden increase in AI capabilities, or a coordinated betrayal. However, we argue that even an incremental increase in AI...

Jan 30, 2025188

Load More (7/10)

LESSWRONG
LW

LESSWRONG
LW

David Duvenaud

David Duvenaud

David Duvenaud

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

Simple probes can catch sleeper agents

Summary of our Workshop on Post-AGI Outcomes

David Duvenaud

Disempowerment patterns in real-world AI usage

When does competition lead to recognisable values?

The Economics of Transformative AI

Upcoming Workshop on Post-AGI Economics, Culture, and Governance

Summary of our Workshop on Post-AGI Outcomes

Upcoming workshop on Post-AGI Civilizational Equilibria

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

Simple probes can catch sleeper agents

Summary of our Workshop on Post-AGI Outcomes

Disempowerment patterns in real-world AI usage

When does competition lead to recognisable values?

The Economics of Transformative AI

Upcoming Workshop on Post-AGI Economics, Culture, and Governance

Summary of our Workshop on Post-AGI Outcomes

Upcoming workshop on Post-AGI Civilizational Equilibria

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development