What happens when a model strategically alters its exploration to resist RL training? In this post, we share our conceptual framework for this threat model, expanding on our research note from last summer. We formalize and decompose "exploration hacking", ahead of an upcoming paper where we study it empirically—by creating...
This is an early-stage research update as part of the ML Alignment & Theory Scholars Program — Summer 2025 Cohort. We welcome any feedback! TL;DR * If misaligned AI realizes it's undergoing RL training that conflicts with its goals, it might be able to sabotage the training by selectively under-exploring....
There have been multiple recent calls for the automation of AI safety and alignment research. There are likely many people who would like to contribute to this space, but would benefit from clear directions for how to do so. Stemming from a recent SPAR project and in light of limitations...
What happens when a model strategically alters its exploration to resist RL training? In this post, we share our conceptual framework for this threat model, expanding on our research note from last summer. We formalize and decompose "exploration hacking", ahead of an upcoming paper where we study it empirically—by creating model organisms that resist RL-based capability elicitation, evaluating detection methods on them, and auditing frontier models' propensity to exploration hack. We welcome feedback on our framework and encourage researchers to explore the open problems we outline.
Huge thanks to David Lindner, Roland Zimmermann and Scott Emmons for their excellent supervision on this project.
Reinforcement learning (RL) is increasingly central to safety training, frontier reasoning... (read 2413 more words →)
This is an early-stage research update as part of the ML Alignment & Theory Scholars Program — Summer 2025 Cohort. We welcome any feedback!
TL;DR
There have been multiple recent calls for the automation of AI safety and alignment research. There are likely many people who would like to contribute to this space, but would benefit from clear directions for how to do so. Stemming from a recent SPAR project and in light of limitations of current systems, we provide a brief list of concrete projects for improving the ability of current and near-future agentic coding LLMs to execute technical AI safety experiments. We expect each of these could be meaningfully developed as short-term (1 week to 3 months) projects.
This is in no way intended to be a comprehensive list, and we strongly welcome additional project ideas in the comments.
Note: Due... (read 2336 more words →)