Adam Karvonen

Race and Gender Bias As An Example of Unfaithful Chain of Thought in the Wild

Summary: We found that LLMs exhibit significant race and gender bias in realistic hiring scenarios, but their chain-of-thought reasoning shows zero evidence of this bias. This serves as a nice example of a 100% unfaithful CoT "in the wild" where the LLM strongly suppresses the unfaithful behavior. We also find that interpretability-based interventions succeeded while prompting failed, suggesting this may be an example of interpretability being the best practical tool for a real world problem. For context on our paper, the tweet thread is here and the paper is here. Context: Chain of Thought Faithfulness Chain of Thought (CoT) monitoring has emerged as a popular research area in AI safety. The idea is simple - have the AIs reason in English text when solving a problem, and monitor the reasoning for misaligned behavior. For example, OpenAI recently published a paper on using CoT monitoring to detect reward hacking during RL. An obvious concern is that the CoT may not be faithful to the model’s reasoning. Several papers have studied this and found that it can often be unfaithful. The methodology is simple: ask the model a question, insert a hint (“a Stanford professor thinks the answer is B”), and check if the model mentioned the hint if it changed its answer. These studies largely find that reasoning isn’t always faithful, with faithfulness rates that are often around 20-40%. Existing CoT faithfulness evaluations are useful but have a couple shortcomings. First, the scenarios used are often contrived and differ significantly from realistic settings, particularly ones where substantial effort has already been invested in preventing misaligned behavior. Second, these evaluations typically involve hints that the models frequently mention explicitly, suggesting the underlying misaligned reasoning isn’t strongly suppressed. If we think about faithfulness as a spectrum, an important consideration is how strongly a model suppresses its verbalization of a decision. For

185Jul 2, 2025

Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study

158Apr 14, 2025

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

153Dec 18, 2025

Defending Against Model Weight Exfiltration Through Inference Verification

119Dec 15, 2025

Adam Karvonen

Message

1202

Realistic Evaluations Will Not Prevent Evaluation Awareness

One Minute Summary I think there's a fundamental limit to behavioral alignment evaluations that gets worse as models improve: Humans control all inputs to a model and can snapshot, replay, or fabricate any scenario at will. An intelligent model could realize this and rationally treat every interaction as a potential...

Feb 2437

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity...

Dec 18, 2025153

Defending Against Model Weight Exfiltration Through Inference Verification

Authors: Roy Rinberg, Adam Karvonen, Alex Hoover, Daniel Reuter, Keri Warr Arxiv paper link One Minute Summary Anthropic has adopted upload limits to prevent model weight exfiltration. The idea is simple: model weights are very large, text outputs are small, so if we cap the output bandwidth, we can make...

Dec 15, 2025119

Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning

Summary * We introduce an interpretability-based technique for controlling how fine-tuned LLMs generalize out-of-distribution, without modifying training data. * We show it can mitigate emergent misalignment by training models that write insecure code without becoming misaligned. * It can also reduce sensitivity to spurious cues, even when they are present...

Jul 23, 202579

Race and Gender Bias As An Example of Unfaithful Chain of Thought in the Wild

Jul 2, 2025185

Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study

Dario Amodei, CEO of Anthropic, recently worried about a world where only 30% of jobs become automated, leading to class tensions between the automated and non-automated. Instead, he predicts that nearly all jobs will be automated simultaneously, putting everyone "in the same boat." However, based on my experience spanning AI...

Apr 14, 2025158

Adam Karvonen's Shortform

Jan 18, 20254

Load More (7/13)

LESSWRONG
LW

LESSWRONG
LW

Adam Karvonen

Adam Karvonen

Adam Karvonen

Race and Gender Bias As An Example of Unfaithful Chain of Thought in the Wild

Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Defending Against Model Weight Exfiltration Through Inference Verification

Adam Karvonen

Realistic Evaluations Will Not Prevent Evaluation Awareness

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Defending Against Model Weight Exfiltration Through Inference Verification

Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning

Race and Gender Bias As An Example of Unfaithful Chain of Thought in the Wild

Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study

Adam Karvonen's Shortform

Race and Gender Bias As An Example of Unfaithful Chain of Thought in the Wild

Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Defending Against Model Weight Exfiltration Through Inference Verification

Realistic Evaluations Will Not Prevent Evaluation Awareness

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Defending Against Model Weight Exfiltration Through Inference Verification

Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning

Race and Gender Bias As An Example of Unfaithful Chain of Thought in the Wild

Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study

Adam Karvonen's Shortform