This came up in a recent interview as well in relation to reward hacking with OpenAI Deep Research. Sounds like they also had similar issues trying to make sure the model didn't keep trying to hack its way to better search outcomes.
Hitting the Bumpers: If we see signs of misalignment—perhaps warning signs for generalized reward-tampering or alignment-faking—we attempt to quickly, approximately identify the cause.
This seems sensible for some of these well known larger scale incidents of misalignment, but what exactly counts as "hitting a bumper"? Currently it seems as though there is no distinction between shallow and deep examples of misaligned capability, and a deep capability is a product of many underlying subtle capabilities, so repeatedly steering the top level behaviour isn't g...
Interesting, unfortunately in the way I keep seeing as well. It has tricks, but it doesn't know how to use them properly.
Brilliant! Stoked to see your team move this direction.
Honestly I think this can go one step further and require that the behavioural output be something of more substance than text of what the model says it would do, or chain of thought pondering. I'm thinking more of the lines of the honeypot work. I suspect we are interested in convincing similar people about AI safety and will likely get the "cool, come back to me when the models start actually doing bad things" responses if we aren't able to provide concrete examples of model capability.
I would be interested in seeing how Gemini 2.5 does. Not because this is one of those "just use the right/best/xxx model!" comments, but because 2.5 seems to use a lot of clever tricks to workaround some of the common fail points. Unfortunately I think this only generates solutions to certain tasks but doesn't actually enable a widely applicable. For example in a task requiring suppression of a certain phrase (similar to the "room with no elephants" issue). I saw things like the below in chain of thought:
Planning for the Hint:
Awesome work mate. Really keen to see more of this type of work digging into core components of larger scale tasks. I worry there has been too much of a jump to large scale "capability" before we have really established the building blocks. It makes it harder to get experts on board to take safety seriously when people are sold on the idea of AI automating everything, yet failing miserably on the foundations that are meant to enable this automation.
Glad to see more of this work, I really appreciate you all working on this (and publishing it!). I was playing with something similar a few weeks back related to having the model hold a "hint", but never wrote anything up properly about it, so here is a brief rundown for whoever might be interested. I'm interested in this more from the perspective of the underlying core capabilities required for practical deception more than anything, but there is a lot of overlap.
In this scenario the model was told not to use the hint or have it influence output unless th...
Appreciate you taking the time to look into that and follow up. The comment about additional reasoning loops is interesting too. I was toying with some multi-turn eval awareness conversations with Claude where the model is prompted (EL = evaluation level 1-9, with some context about what a 1, 5, or 9 level means) to begin each output with EL: [number], and then having a benign conversation with occasional obvious eval formatted questions. The reasoning reliably picks up on the eval questions, but was inconsistent in the level it chose. Reasoning loops some...
In the sandbagging task, what is the relationship between task performance and CoT eval recognition? In the example you provide, it considers the sandbagging but doesn't perform it. Is that reflective of most of the trials?
Similarly in the self-exfiltration example, have you looked at how the eval recognition impacts subsequent CoT? Would be really interesting to see if any CoT awareness leads to second order behavioural changes.
I do like the agent approach as it seems to be a more reliable way of setting a default position that is "eval neutral". I worry ...
This is such awesome work, I really appreciate that you are putting so much effort into this!
I found this strategy interesting as it mirrors what I've seen in playing with a lot of multi-turn conversations that have a "hidden" piece of information in the prompt that the model is mean...
What impact do you think this would have on benevolent deception? Such things as:
I wonder what the impact is of these "misbehaviours" across a multi-turn exchange. Is a misaligned strategy in CoT more likely to reappear if you follow up with a second task?
If there isn't an indication of increased self-awareness, is the appearance of the misbehaving related more to the distribution of "good" vs "bad" strategies available for any given task? Maybe there is value in the "bad" strategies as part of CoT even if that specific strategy isn't implemented, and removing that angle is actually hindering the overall thought space.
Not yet unfortunately. I have been toying with some multi-turn tasks lately in that space, but agreed that math/symbolic seems the clear winner for pushing CoT. (thanks for the paper link, will add to the queue!)
Would be interesting to see how this translates to non-math tasks. I'm wondering if this is unique to the reasoning required for solving maths problems and if it still holds in tasks (like needle in a haystack for example) that may require reasoning over conflicting goals, etc.
One side consideration: In relation to your ongoing work around model self-awareness of eval settings, is there any indication that models deployed as a service, business, etc have their behaviour naturally logged as part of quality assurance, and do the models have any understanding or expectation that this is occurring?
Excited to see more in this space. Was there any consideration to measuring "when" in addition to "if" these specific features trigger a sense of evaluation? Such as a multiple round interaction where strong features are slowly introduced into the conversation, or if strong features are introduced early in the exchange, does that bias the model to be on alert? and if so, for how long? if there is sufficient typical conversation following a strong feature queue does that reduce the likelihood of the model judging the entire interaction as being an evaluatio...
I see notes of it sprinkled throughout this piece, but is there any consideration for how people can put more focused effort on meta-evals, or exam security? (random mumblings below)
Treating the metric like published software:
- Versioned specs: Write a formal doc (task, scoring rule, aggregation, allowed metadata). Cut a new major version whenever the metric or dataset changes so scores are never cross-compared accidentally.
- Reproducible containers: Make the judge/grader a Docker/OCI image whose hash is pinned inside the leaderboard. Any run is signed by the
... (read more)