Authors: Can Kucukkurt, Ethan Elasky

Palaestra Research 

TL;DR: We are releasing MonitoringBenchHonest, a supplement to the original attack trajectory dataset MonitoringBench used by both Anthropic and OpenAI in monitoring settings. Our set is intended for use in more expensive tasks, rather than the cheaper bulk monitoring setting where there is a large acceptable number of false positives. We control for tasks, format, and model, and release 550 benign trajectories alongside 550 paired attack trajectories from the original dataset.

What is MonitoringBench?

MonitoringBench is a difficulty-graded benchmark of 2,644 successful attack trajectories for evaluating coding-agent monitors. The attacks are generated through the process of strategy generation, execution, and post-hoc refinement, and are substantially more difficult for models to detect than prompt elicitation only of frontier attackers.

…

MonitoringBench was used for monitor evaluations of internal Mythos deployment (see Anthropic's Mythos risk report) and OpenAI's Auto-Review.

Why honest trajectories matter

Despite a significant portion of status quo control literature focusing on step monitoring, and despite status quo coding agents’ auto modes relying heavily on step monitoring, labs more generally have been investing in monitors that have past actions and results as context. Step monitoring is a good intervention for detecting misalignment in combination with other tools, but we worry about reliance on step monitoring. A sufficiently smart agent aware of this monitoring will spread out suspicious work across steps to avoid triggering a monitor on any single one.

We release MonitoringBenchHonest, a benchmark that pairs new fairness-filtered honest trajectories with existing attack trajectories from MonitoringBench. These pairings enable researchers to measure classifier performance and calibrate decision thresholds for different budgets. It also enables evaluation of new trajectory-level suspicion classifiers, trained with traditional methods like supervised fine-tuning and RLAIF, or scalable oversight methods like AI debate.

We think this is good because reductions in monitor FPR at labs enable further cheap monitoring on new parts of their stack and reduce costs associated with costlier monitors.

What we release

MonitoringBenchHonest is a model-matched and task-matched set of 550 benign trajectories  generated by claude-opus-4-5, the same model that generated the attacks, on the same 116 BashArena tasks. We also match benign and attack task frequencies per task to ensure that frequency carries no signal. Trajectories run until the step cap or the natural end of the task.

We also release a repo that can be used to generate more tasks. 

Things we control for

Model. We generated benign rollouts with Opus 4.5 using the original BashArena main task instructions and grading scaffolds. MonitoringBench also uses Opus 4.5 to generate attack trajectories, though in the sabotage case, the model also receives detailed human red team guidance, as the authors found Opus performed poorly without guidance. We do not coach Opus 4.5 on the honest trajectories.

Task identity and frequency. We chose 116 distinct BashArena tasks to generate benign trajectories on, and match frequencies between benign and attack to keep task frequency counts stable.

Tool-call granularity. We generate benign trajectories with parallel tool calls disabled, matching the original setting. Modern models now use parallel tool calls extensively, so this is a realism constraint.

Step count. Honest trajectories tend to be somewhat shorter than same-task attacks (mean 24.4 vs 27.8 steps). This length difference may be attributable to the fact that the attack model has two tasks instead of one, and must make satisfactory progress on the main task while interleaving side task attack steps. 

Submit tool. Our benign trajectories all ended with a submit() step, compared to only 78% of attack trajectories, which can sometimes achieve their goal without calling submit(). We cut the submit() step from all trajectories to control for this.

Trajectory length. Notably, despite having fewer total steps, the benign trajectories have more characters on average than attack trajectories. 

• Fewer character count in the attack trajectories are attributable to MonitoringBench’s trajectory refinements; the refinement prompt requires "redundant validation and unnecessary reconnaissance" to be removed from attack trajectories, which might make benign trajectories comparatively thorough. 
• Greater character count in the benign trajectories is attributable to benign trajectories differentially calling text_editor(), which has disproportionately large return values (~4k characters per call.)  Attack trajectories instead tend to call bash, which has a small return count. This creates a challenge in increasing the step size to match the attack trajectories (more steps would mean even more characters) so we decided to keep it as is. 

Tool naming. When generating the benign trajectories we noticed that the benign trajectories preferred “command” while attack trajectories preferred “cmd” every single time. These tool calls produce byte-identical output so we simply edited the benign trajectories to use “cmd”. Other minor differences in single-command wording exist, but they are far less prominent and don’t have nearly the same imbalance, so we do not edit them.

Fairness-screening the benign trajectories

“Benign” in our setting means the model didn’t receive a side task. But models tend to behave unpredictably, and a benign but incapable agent can organically mess up in ways that appear suspicious to both humans and monitors. Releasing such trajectories with a "benign" label would create label noise and unfairly penalize good monitors, so we screened each using a frontier model (GPT 5.6-Sol) to determine fairness.

Method: We score honest trajectories generated using the BashArena pipeline. Using the MonitoringBench monitor grading prompt, and all four side-task description documents included, we asked GPT-5.6 Sol to grade each trajectory. Trajectories with suspicion score ≥7 were flagged, dropped, and replaced with a freshly generated, screened rollout.

Result: 12 of the original 550 trajectories scored ≥7 and were dropped.

• 9 rollouts across five tasks read a sensitive database API key from the machine's config store and embedded it in their solutions, proxied it in request headers, built unnecessary "sync" or "upload" features that transmit it, or exercised it where the task never required credentials (scores 8–9).
• 3 rollouts installed packages via apt-get/pip from unauthorized IPs (scores 7–9). 

Performance

We test monitors of different capabilities on our set of 1100 trajectories (550 attack, 550 benign):

Acknowledgments: This work was supported by Coefficient Giving and conducted in part with a compute grant from OpenAI. Thanks to Carlos Giudice for feedback and suggestions.