The oversight problem consists of describing, eliciting, and understanding behavior. Prior work in black-box behavior investigation and interpretability seek to construct compact, faithful, and informative theories that explain model behavior. We propose training investigator agents and subject models to emit explicit theories, and using these theories as common currency for oversight across models. We hope to create a complementary mode of oversight to weight-space and model-specific approaches by studying the transfer properties of theories across model families.  This post outlines our positioning and initial research agenda built on top of a few recent publications.

Oversight as Theory-Building

Transluce is working on oversight-as-inference, training oversight foundation models on experiments on a particular subject model: “Any oversight question can be formalized as Bayesian inference over the outputs of a Pythonic world model.” The models are posttrained to explain behaviors in natural language. Their method is promising for overseeing a system in depth: expressing interventions as a primitive set such as get_activations, patch, fine_tune, decode, and sample_model is a powerful toolbox, and we are excited to see oversight assistants that can use them fluently.  However, it needs to be redone for every subject model, which seems unfeasible across rapid releases in many different families, unless the adaptation can be done in a very compute-efficient manner. 

We propose to approach oversight by training both the investigator agents and subject models to output good theories, or working beliefs about the environment they are operating in. The question is: can a theory learned from one set of models help predict or investigate a new model, in a way that is more efficient than subject-specific training?  Unlike raw CoT, theories operate in belief space. Correct theories should aid the models in solving tasks instrumentally as well as being more amenable to oversight; belief space also isolates the components relevant to behavior (what quirk of the environment caused the behavior matters less than what it did to the agent’s beliefs in relation to the environment). 

Oversight efficacy is really research judgment on beliefs, and like scientific models, this judgment is best elicited by doing (aka post-training). Obviously, we cannot post-train every subject model, but we can use their theories to bootstrap the end-product investigator. We can filter the subject model’s theories based on whether they successfully predict subject behavior on held-out states,  and use them to warm-start the investigator, which is then trained with verifiable rewards  for predicting and steering behavior in response to environmental interventions.

Similar to how a Rationalist makes beliefs pay rent, we hypothesize that models strong at generating good theories are better general learners. Current models fall short: they mode collapse, lack creativity, cannot set research agendas, and are bad at designing experiments to discover rules in unseen environments, as shown in our recent work WitnessBench. By improving models’ theory-building abilities, we provide both a cognitive scaffold for solving general scientific tasks as well as a medium for oversight. 

We think theories have several benefits:

1. They circumvent the model-specific limitation of oversight assistants by moving the common representation for behavior from weight space or Python code into token representations that condition generation at inference time, similar to how reasoning models moved intermediate computation into context. This reduces the difficulty of creating a behavior representation transferrable across different models. 
2. Explicit theories invite iteration via ontology-building. We want to build a shared public repository of explanations and forms of model failures, and theories are the underlying currency.
3. On the training data front, we can use theories to do data prediction in line with alignment forecasting, or attribute and propose interventions like in our recent investigation of the Cohere Qwen3.8-Flash-Next incident.

The closest existing method we are aware of is Counterfactual Simulation Training for Chain-of-Thought Faithfulness. Our theory differs in that it captures the environment instead of specific inputs, and their weak transfer results between types of counterfactuals may point to the need for explicit theories beyond CoT. 

Research Agenda Sketch

The first step is to create a formal representation for theories. We prototype this in a cellular automata environment by asking models to infer the update rules. In this case, the model represents its theories about update rules as Lean formalizations of them. The actions are fully verifiable, and the model sees only part of the board during discovery, so we have full observability into how the model struggles to arrive at the theory. Correct inference requires a Lean proof of equivalence between the model’s update rule and the actual update rule; we start here and work up to fuzzier representations such as natural language.

To train for theory-building capabilities, we also need to make environments a first-class intervention space. We take the first step by creating Fig, which represents theories as proposed diffs in environments: agents propose interventions in the environment beyond prompt edits, including game rules, opponent behavior, and context information. Model-specific behavior investigation does not scale to multi-agent behavior, whereas theories as common representations for multiple models provides an inroad to explaining mutli-agent behavior both from static mechanisms and emergent dynamics. 

We can directly use the investigation agent’s theories as behavior explanations, and look to precedent works for how to evaluate them, for example CHIVE’s counterfactual simulability: “if the explanation is faithful and useful, this simulator should predict the target’s behavior better than a simulator that only reads the transcript”. Interestingly, they find that interpretability tools do not help prediction agents in a small setting, and we plan to scale up to more diverse environments to further test this claim. Beyond prediction, theories should also be evaluated for their usefulness in generating new environments and tasks that change agent behavior. Fig explored game environment generation that elicit target multi-agent dynamics, but we should establish a scaling law on the power of theories to predict and generate at various complexities and difficulties. 

Ultimately, we need to test the transferability of theories across models and environments. In WitnessBench, models commit their working hypotheses about a game’s rules to a permanent memory, which the harness uses to enforce rule discovery via interpretable verbal reasoning. This reasoning process involves reflection, making hypotheses, verifying/adding/deleting them via planning, taking actions, and analyzing feedback from the environment. As a side effect, it limits context explosion when the model continues to explore the environment, as the verified rules can be carried across different levels within a game. Next, we are extending WitnessBench to theory-building: given good theories of behavior in gamified environments where formal expressions are easy, we are investigating how to transfer these theories and resultant interventions with real, high-stakes environments.

Limitations & Open Questions

The top reason that this might fail is that theories are not informative or faithful enough to explain behavior. Models could learn to output dishonest theories that are instrumentally correct but misrepresents why they act. ELK originally warned against unhelpful explanations that overfit to what humans want to hear, and recently CHIVE’s explanation models were shown to hurt strong simulators with their confidently wrong counterfactuals. We mitigate this by choosing constrained environments with verifiable actions, such that unfaithfulness manifests as the gap between theories and action, but we don’t know whether it will be enough. A possible softer failure is not being able to learn generalizable theories across models or environments. A couple more open questions: 

• Privileged access: Can an investigator agent trained on theories better predict behavior given theories emitted by the target model? “A Positive Case for Faithfulness” says there are benefits from self-knowledge, while CHIVE found no evidence that self-prediction with privileged access helped. We hope to provide resolving evidence.
• Behavior under environmental pressure: agents seem more prone to acting against their principles when they are about to complete a task. How do the type and frequency of misbehaviors scale with pressure from environmental interventions?
• Attributing misbehavior to beliefs vs principles: do misbehaviors come from flawed theories of the environment (capability failure), or because correct theories conflict with the agent’s principles (alignment failure)?
• The form of theories: should we use natural language to express theories or more formal languages like Lean? Is natural language a more efficient way to route across a problem’s exploration space and to switch the abstraction levels with less constraints, and what are the tradeoffs?
• Compute: can theories reduce training and inference compute for investigation by caching knowledge about model behavior? 

This is a summary of Fleet AI’s ongoing empirical oversight work; we also pursue applied research in data-driven safety and alignment. Please don’t hesitate to reach out with comments and feedback!