Marc Carauleanu

Self-Other Overlap: A Neglected Approach to AI Alignment

Figure 1. Image generated by DALL·E 3 to represent the concept of self-other overlap Many thanks to Bogdan Ionut-Cirstea, Steve Byrnes, Gunnar Zarnacke, Jack Foxabbott and Seong Hah Cho for critical comments and feedback on earlier and ongoing versions of this work. This research was conducted at AE Studio and supported by the AI Safety Grants programme administered by Foresight Institute with additional support from AE Studio. Summary In this post, we introduce self-other overlap training: optimizing for similar internal representations when the model reasons about itself and others while preserving performance. There is a large body of evidence suggesting that neural self-other overlap is connected to pro-sociality in humans and we argue that there are more fundamental reasons to believe this prior is relevant for AI Alignment. We argue that self-other overlap is a scalable and general alignment technique that requires little interpretability and has low capabilities externalities. We also share an early experiment of how fine-tuning a deceptive policy with self-other overlap reduces deceptive behavior in a simple RL environment. On top of that, we found that the non-deceptive agents consistently have higher mean self-other overlap than the deceptive agents, which allows us to perfectly classify which agents are deceptive only by using the mean self-other overlap value across episodes. Introduction General purpose ML models with the capacity for planning and autonomous behavior are becoming increasingly capable. Fortunately, research on making sure the models produce output in line with human interests in the training distribution is also progressing rapidly (eg, RLHF, DPO). However, a looming question remains: even if the model appears to be aligned with humans in the training distribution, will it defect once it is deployed or gathers enough power? In other words, is the model deceptive? We introduce a method that aims to reduce deception and increas

243Jul 30, 2024

Marc Carauleanu

Message

AI Safety Researcher @AE Studio

Currently researching a neglected prior for cooperation and honesty inspired by the cognitive neuroscience of altruism called self-other overlap in state-of-the-art ML models.

Previous SRF @ SERI 21', MLSS & Student Researcher @ CAIS 22' and LTFF grantee.

700

136

Mistral Large 2 (123B) seems to exhibit alignment faking

UPDATE: Recent work with improved AF and compliance gap classifiers disagrees with our results. We recommend using the improved classifiers. Summary We wanted to briefly share an early takeaway from our exploration into alignment faking: the phenomenon appears fairly rare among the smaller open-source models we tested (including reasoning models)....

Mar 27, 202581

Reducing LLM deception at scale with self-other overlap fine-tuning

This research was conducted at AE Studio and supported by the AI Safety Grants program administered by Foresight Institute with additional support from AE Studio. Summary In this post, we summarize the main experimental results from our new paper, "Towards Safe and Honest AI Agents with Neural Self-Other Overlap", which...

Mar 13, 2025162

Self-Other Overlap: A Neglected Approach to AI Alignment

Jul 30, 2024243

The 'Neglected Approaches' Approach: AE Studio's Alignment Agenda

Many thanks to Samuel Hammond, Cate Hall, Beren Millidge, Sumner Norman, Steve Byrnes, Lucius Bushnaq, Joar Skalse, Kyle Gracey, Gunnar Zarncke, Ross Nordby, David Lambert, Simeon Campos, Bogdan Ionut-Cirstea, Ryan Kidd, and Eric Ho for critical comments and suggestions on earlier drafts of this agenda, as well as Philip Gubbins,...

Dec 18, 2023190

Towards empathy in RL agents and beyond: Insights from cognitive science for AI Alignment

This is a talk I gave at the recent AI Safety Europe Retreat (AISER) on my research on obtaining insights from the cognitive science of empathy and applying them to RL agents and LLMs. Talk link: https://clipchamp.com/watch/6c0kTETRqBc Slides link: https://bit.ly/3ZFmjN8 Talk description: I begin by presenting a short review on...

Apr 3, 202315

Lessons learned and review of the AI Safety Nudge Competition

TL;DR: We ran a competition to encourage people to stop procrastinating on their AI Safety projects. We had 76 applicants out of which over 40% (31 participants) completed their goal by the end of October and were added to a draw to win a monetary prize. The vast majority of...

Jan 17, 20233

Winners of the AI Safety Nudge Competition

The AI Safety Nudge Competition aims to encourage people to do things related to AI Safety today instead of procrastinating it into the future by allowing them to enter into a draw if they complete the goal that they set for themselves. Participants defined a specific goal for themselves, examples...

Nov 15, 20224

Load More (7/9)

LESSWRONG
LW

LESSWRONG
LW

Marc Carauleanu

Marc Carauleanu

Marc Carauleanu

Self-Other Overlap: A Neglected Approach to AI Alignment

The 'Neglected Approaches' Approach: AE Studio's Alignment Agenda

Reducing LLM deception at scale with self-other overlap fine-tuning

Mistral Large 2 (123B) seems to exhibit alignment faking

Marc Carauleanu

Mistral Large 2 (123B) seems to exhibit alignment faking

Reducing LLM deception at scale with self-other overlap fine-tuning

Self-Other Overlap: A Neglected Approach to AI Alignment

The 'Neglected Approaches' Approach: AE Studio's Alignment Agenda

Towards empathy in RL agents and beyond: Insights from cognitive science for AI Alignment

Lessons learned and review of the AI Safety Nudge Competition

Winners of the AI Safety Nudge Competition

Mistral Large 2 (123B) seems to exhibit alignment faking

Reducing LLM deception at scale with self-other overlap fine-tuning

Self-Other Overlap: A Neglected Approach to AI Alignment

The 'Neglected Approaches' Approach: AE Studio's Alignment Agenda

Towards empathy in RL agents and beyond: Insights from cognitive science for AI Alignment

Lessons learned and review of the AI Safety Nudge Competition

Winners of the AI Safety Nudge Competition

Self-Other Overlap: A Neglected Approach to AI Alignment

The 'Neglected Approaches' Approach: AE Studio's Alignment Agenda

Reducing LLM deception at scale with self-other overlap fine-tuning

Mistral Large 2 (123B) seems to exhibit alignment faking