Automating subtle emergent misalignment research

evhub; Oscar Obeso; rqi; Monte M

Automating subtle emergent misalignment research — LessWrong

41 Automating subtle emergent misalignment research

by evhub, Oscar Obeso, rqi, Monte M

3rd Nov 2026

AI Alignment Forum

1 min read

41 Ω 17

This is a linkpost for https://alignment.anthropic.com/2026/automated-em-search

Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger

Abstract

Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for this game and the resulting models never deployed, the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior. Examples like these could represent a possible gap in current training monitoring defenses and could be used to test and improve future monitors. We see this work as a first step toward automated adversarial stress-testing of our safety systems where a safety question can be turned into a score that autonomous agents can then help solve, and are now applying similar approaches to other safety questions.

Twitter thread

New Anthropic research: Automating subtle emergent misalignment research.

Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor.

These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior.

Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute.

41 Ω 17

Automating subtle emergent misalignment research

New Comment

5 comments, sorted by

top scoring

Click to highlight new comments since: Today at 8:04 AM

[-]draganover12hΩ72211

Thanks! I'm pretty surprised that this seems to be a similar hill-climbing scaffold to the one from the automated weak-to-strong research post ~6 months ago? In particular, I find the following two things intrinsically at odds:

Bullishness on automated research being on the critical path to alignment
Prioritizing hill-climbing settings for automated alignment research (at least publicly)

There is a belief among large parts of the 3rd party safety community that alignment requires the kind of research which cannot easily be hillclimbed. I.e., it is not always clear what metrics one should climb. And, even if one obtains good scores on the metrics, it's not clear how this corresponds to improved alignment. I'm curious whether you disagree with this assessment?

To ground this in specific examples, we've used similar scaffolds to the one described in this post. While these can be useful for hill-climbing-shaped work, we find that they do not provide uplift on the diverse set of tasks that holistic alignment research seems to require.

[-]evhub12hΩ89-17

I used to be a big proponent of the position that automating alignment would require fuzzy alignment taste that we would have to train into models explicitly, but I've recently updated away from that position and I think it now looks much more likely to me that hillclimbing concrete metrics will be more important. I'll probably write more about this soon, but for now:

I think this research provides a very good example of how we can set up a concrete hillclimbable task and then get useful and interesting insights out the other end, not just number going up in an inscrutable way. If you look at the data here, a lot of it does make sense once you think about it, e.g. making use of implicit power-seeking vibes in descriptions of evolutionary processes.
I also just generally think way more different alignment objectives can be converted into auditing games than is currently appreciated. Auditing games are really powerful! E.g. I think all of interpretability can be made concrete that way. And all you need to make that work is an automated pipeline for generating model organisms, which is exactly what this research is!

I also just really disagree with the idea that we should be putting effort into scaffolds; I think the returns to better scaffolds, at least relative to what the labs already have, are pretty minimal, and tend to get repeatedly washed out by improvements in the underlying models. Working on scaffolds is just not a very bitter-lesson-pilled thing to do; in my opinion, the right approach is to focus on metrics and verification processes, which scale much better, and is what we did here.

[-]Dean Valentine12h*168

Isn't the primary requirement of "hill climbing" on alignment that you have some kind of oracle that tells you whether or not a model is "misaligned"? There are already example environments showing production Opus doing antisocial things. How do you quantify how well Anthropic did with the Opus 5.5 release, much less how this red teamer did?

Perhaps the blog answers this and I am just committing the cardinal sin of not reading it before commenting, but it seems like the main problem with any strategy to "hill climb" to safe AGI, is that there's no way to assess the alignment of arbitrary models, even ones "only" as smart as Opus 5.5.

[-]Bronson Schoen4h53

It seems like all of the following can be true:

This research sets up a hill-climbable task which does produce serious mitigations to data poisoning as a threat model
Way more different alignment objectives can be converted into auditing games than is currently appreciated
Hillclimbing concrete metrics is more important^[1] than the issues raised in Automating Alignment Is Harder Than You Think (ex: Scalable Oversight - 4.2.2 Correlated uncertainty makes scalable oversight for OSAs harder).

Yet that still doesn't avoid the need to solve W2S/SO. There's work like Automated Weak-to-Strong Researcher where the automation is targeting W2S/SO itself, but this would be at best targeting that indirectly?

I think all of interpretability can be made concrete that way. And all you need to make that work is an automated pipeline for generating model organisms

Qualitatively while this would give you a lot of different model organisms, I'm skeptical that they're diverse in the way that would make them interesting as a "zoo" to perform interpretability on to understand something like powerseeking, reward-seeking, etc, as I'd imagine they're much more persona based and more importantly those produced by a single training method seem to in practice often be susceptible to interpretability techniques that don't generalize across training methods. I could imagine though an idea where you extend this pipeline over time to allow the red team to use a more diverse range of training methods, then your "generator" for model organisms grows more diverse over time while still leveraging increasing model capabilities^[2], which seems pretty promising^[3].

^{^}
Hard to operationalize but if we imagine something like "you could only do one"
^{^}
Modulo elicitation
^{^}
I'd imagine now / soon models on red team are capable enough to mix data / training interventions in truly cursed ways that would've taken a lot of researcher time to think of

[-]draganover11h42

Interesting! looking forward to reading the further thoughts once they’re ready. Agreed regarding the bitter-lesson pilled perspective on scaffold engineering

Moderation Log