Simon Fischer

How to safely use an optimizer

Summary: The post describes a method that allows us to use an untrustworthy optimizer to find satisficing outputs. Acknowledgements: Thanks to Benjamin Kolb (@benjaminko), Jobst Heitzig (@Jobst Heitzig) and Thomas Kehrenberg (@Thomas Kehrenberg) for many helpful comments. Introduction Imagine you have black-box access to a powerful but untrustworthy optimizing system, the Oracle. What do I mean by "powerful but untrustworthy"? I mean that, when you give an objective function f as input to the Oracle, it will output an element x that has an impressively low[1] value of f(x). But sadly, you don't have any guarantee that it will output the optimal element and e.g. not one that's also chosen for a different purpose (which might be dangerous for many reasons, e.g. instrumental convergence). What questions can you safely ask the Oracle? Can you use it to create utopia by asking for designs of machines, proteins, computer programs, etc.? Or are you risking the destruction of everything that we value if you dare to use such designs? As an example, the Oracle could be a learned system; in that case, the topic of this post would be finding a way to get useful work out of the Oracle despite its inner misalignment. In this post I'll describe a technique that allows us to safely use the Oracle under fairly weak assumptions. This approach can also be considered to be a way of controlling arbitrarily powerful AI systems. Edit: I've written a bit more on the motivation for this setting in a comment. One neat trick > This isn't fair, isn't fair, isn't fair! There's a limit to how many constraints you can add to a problem before it really is impossible! > > (Harry Potter and the Methods of Rationality, Chapter 56) Let O be a finite set of possible outputs of the Oracle (e.g. strings of length at most l) and f:O→R be our objective function. Let's assume we are happy with an output that satisfices; i.e. we want to find an output x such that the value of f(x) is lower than s

47Mar 28, 2024

Simon Fischer

Message

490

101

16y

[Aspiration-based designs] 2. Formal framework, basic algorithm

Summary. In this post, we present the formal framework we adopt during the sequence, and the simplest form of the type of aspiration-based algorithms we study. We do this for a simple form of aspiration-type goals: making the expectation of some variable equal to some given target value. The algorithm...

Apr 28, 202418

[Aspiration-based designs] 1. Informal introduction

Sequence Summary. This sequence documents research by SatisfIA, an ongoing project on non-maximizing, aspiration-based designs for AI agents that fulfill goals specified by constraints ("aspirations") rather than maximizing an objective function . We aim to contribute to AI safety by exploring design approaches and their software implementations that we believe...

Apr 28, 202444

How to safely use an optimizer

Mar 28, 202447

AISC team report: Soft-optimization, Bayes and Goodhart

This is a report on our work in AISC Virtual 2023. For AISC 2023, our team looked into the foundations of soft optimization. Our goal at the beginning was to investigate variations of the original quantilizer algorithm, in particular by following intuitions that uncertainty about goals can motivate soft optimization....

Jun 27, 202338

Pivotal acts using an unaligned AGI?

Introduction A recent post (AGI Ruin: A List of Lethalities) gives an argument for humanity’s likely inability to survive the invention of superintelligent AGI. The argument goes very roughly like this: 1. We haven’t solved the alignment problem (citation needed?). 2. Given access only to an unaligned AI, we can’t...

Aug 21, 202228

The Hammer and the Mask - A call to action

Epistemic status: Pretty confident, seems obviously correct. Summary: We should promote widespread use of reusable silicone masks to stop the spread of SARS-CoV-2. Masks and transmission of SARS-CoV-2 SARS-CoV-2 spreads mainly through respiratory droplets. Most of the discussion of mask usage to prevent transmission has so far been centered on...

Apr 19, 202018

LESSWRONG
LW

LESSWRONG
LW

Simon Fischer

Simon Fischer

Simon Fischer

How to safely use an optimizer

[Aspiration-based designs] 1. Informal introduction

AISC team report: Soft-optimization, Bayes and Goodhart

Pivotal acts using an unaligned AGI?

Simon Fischer

[Aspiration-based designs] 2. Formal framework, basic algorithm

[Aspiration-based designs] 1. Informal introduction

How to safely use an optimizer

AISC team report: Soft-optimization, Bayes and Goodhart

Pivotal acts using an unaligned AGI?

The Hammer and the Mask - A call to action

How to safely use an optimizer

[Aspiration-based designs] 1. Informal introduction

AISC team report: Soft-optimization, Bayes and Goodhart

Pivotal acts using an unaligned AGI?

[Aspiration-based designs] 2. Formal framework, basic algorithm

[Aspiration-based designs] 1. Informal introduction

How to safely use an optimizer

AISC team report: Soft-optimization, Bayes and Goodhart

Pivotal acts using an unaligned AGI?

The Hammer and the Mask - A call to action