sbenthall

Introduction to Towards Causal Foundations of Safe AGI

By Tom Everitt, Lewis Hammond, Rhys Ward, Ryan Carey, James Fox, Sebastian Benthall, Matt MacDermott and Shreshth Malik representing the Causal Incentives Working Group. Thanks also to Toby Shevlane, MH Tessler, Aliya Ahmad, Zac Kenton, Maria Loks-Thompson, and Alexis Bellot. Over the next few years, society, organisations, and individuals will face a number of fundamental questions stemming from the rise of advanced AI systems: * How to make sure that advanced AI systems do what we want them to (the alignment problem)? * What makes a system safe enough to develop and deploy, and what constitutes sufficient evidence of that? * How do we preserve our autonomy and control as decision making is increasingly delegated to digital assistants? A causal perspective on agency provides conceptual tools for navigating the above questions, as we’ll explain in this sequence of blog posts. An effort will be made to minimise and explain jargon, to make the sequence accessible to researchers from a range of backgrounds. Agency First, with agent we mean a goal-directed system that acts as if it it is trying to steer the world in some particular direction(s). Examples include animals, humans, and organisations (more on agents in a subsequent post). Understanding agents is key to the above questions. Artificial agents are widely considered the primary existential threat from AGI-level technology, whether they emerge spontaneously or through deliberate design. Despite the myriad risks to our existence, highly capable agents pose a distinct danger, because many goals can be achieved more effectively by accumulating influence over the world. Whereas an asteroid moving towards earth isn’t intending to harm humans and won’t resist redirection, misaligned agents might be distinctly adversarial and active threats. Second, the preservation of human agency is critical in the approaching technological transition, for both individuals and collectives. Concerns have already been raised

73Jun 12, 2023

sbenthall

Message

373

13y

Reward Hacking from a Causal Perspective

Post 4 of Towards Causal Foundations of Safe AGI, preceded by Post 1: Introduction, Post 2: Causality, Post 3: Agency, and Post 4: Incentives. By Francis Rhys Ward, Tom Everitt, Sebastian Benthall, James Fox, Matt MacDermott, Milad Kazemi, Ryan Carey representing the Causal Incentives Working Group. Thanks also to Toby...

Jul 21, 202329

Incentives from a causal perspective

Post 4 of Towards Causal Foundations of Safe AGI, preceded by Post 1: Introduction, Post 2: Causality, and Post 3: Agency. By Tom Everitt, James Fox, Ryan Carey, Matt MacDermott, Sebastian Benthall, and Jon Richens, representing the Causal Incentives Working Group. Thanks also to Toby Shevlane and Aliya Ahmad. “Show...

Jul 10, 202327

Causality: A Brief Introduction

Post 2 of Towards Causal Foundations of Safe AGI, see also Post 1 Introduction. By Lewis Hammond, Tom Everitt, Jon Richens, Francis Rhys Ward, Ryan Carey, Sebastian Benthall, and James Fox, representing the Causal Incentives Working Group. Thanks also to Alexis Bellot, Toby Shevlane, and Aliya Ahmad. Causal models are...

Jun 20, 202349

Introduction to Towards Causal Foundations of Safe AGI

Jun 12, 202373

Don't Fear the Reaper: Refuting Bostrom's Superintelligence Argument

I've put a preprint up on arXiv that this community might find relevant. It's an argument from over a year ago, so it may be dated. I haven't been keeping up with the field much since I wrote it, so I welcome any feedback especially on where the crux of...

Mar 1, 20179

Autonomy, utility, and desire; against consequentialism in AI design

For the sake of argument, let's consider an agent to be autonomous if: * It has sensors and actuators (important for an agent) * It has an internal representation of its goals. I will call this internal representation its desires. * It has some kind of internal planning function that...

Dec 3, 20147

more on predicting agents

Suppose you want to predict the behavior of an agent. I stand corrected. To make the prediction, as a predictor you need: * observations of the agent * the capacity to model the agent to a sufficient degree of accuracy "Sufficient accuracy" here is a threshold on, for example, KL...

Nov 8, 20141

Load More (7/14)

LESSWRONG
LW

LESSWRONG
LW

sbenthall

sbenthall

sbenthall

Introduction to Towards Causal Foundations of Safe AGI

Causality: A Brief Introduction

Reward Hacking from a Causal Perspective

Incentives from a causal perspective

sbenthall

Reward Hacking from a Causal Perspective

Incentives from a causal perspective

Causality: A Brief Introduction

Introduction to Towards Causal Foundations of Safe AGI

Don't Fear the Reaper: Refuting Bostrom's Superintelligence Argument

Autonomy, utility, and desire; against consequentialism in AI design

more on predicting agents

Introduction to Towards Causal Foundations of Safe AGI

Causality: A Brief Introduction

Reward Hacking from a Causal Perspective

Incentives from a causal perspective

Reward Hacking from a Causal Perspective

Incentives from a causal perspective

Causality: A Brief Introduction

Introduction to Towards Causal Foundations of Safe AGI

Don't Fear the Reaper: Refuting Bostrom's Superintelligence Argument

Autonomy, utility, and desire; against consequentialism in AI design

more on predicting agents