Replying toIntegrity in AI Governance and Advocacy

I agree with all of these (except never felt worried about being quoted by Conjecture)

habryka

Ok, so we both had some feelings about the recent Conjecture post on "lots of people in AI Alignment are lying", and the associated marketing campaign and stuff.

I would appreciate some context in which I can think through that, and also to share info we have in the space that might help us figure out what's going on.

I expect this will pretty quickly cause us to end up on some broader questions about how to do advocacy, how much the current social network around AI Alignment should coordinate as a group, how to balance advocacy with research, etc.

OliviaJ

Feelings about Conjecture post:

Lots of good points about people not stating their full beliefs messing

... (read 6762 more words →)

135

•••

Winners of AI Alignment Awards Research Contest

Orpheus16

Orpheus16, Olive Branch

This post describes the winning submissions of AI Alignment Awards. We offered prizes for novel contributions to the problems of goal misgeneralization and corrigibility. (For more context, see this post and our site.)

Summary

We received 118 submissions. Of these, 5 received honorary mention prizes ($1000) and 7 won final prizes ($5000-16000). The rest of the post summarizes the 7 winning submissions, including comments from our judges. (You can see winners’ full submissions on our site.)

Goal Misgeneralization winners

Thane Ruthenis ($11,000)

Abstract

Goal misgeneralization primarily happens because the system under training latches not upon the goal it's being trained for, but upon an upstream correlate of that goal — like human love for their children is an upstream correlate of... (read 3448 more words →)

115

Shah (DeepMind) and Leahy (Conjecture) Discuss Alignment Cruxes

Olive Branch

Olive Branch, Rohin Shah, Connor Leahy, Andrea_Miotti

Preface

In December 2022, Rohin Shah (DeepMind) and Connor Leahy (Conjecture) discussed why Leahy is pessimistic about AI risk, and Shah is less so. Below is a summary and transcript.

Summary

Leahy expects discontinuities - capabilities rapidly increasing and behavior diverging far from what we aim towards - to be a core alignment difficulty. While this concept is similar to the sharp left turn (SLT), Leahy notes he doesn’t like the term SLT. SLT seems to imply deception or a malicious treacherous turn, which Leahy doesn’t see as an interesting, relevant, or necessary part.

Shah suggests they start by discussing Leahy’s SLT views.

Leahy explains he expects there to be some properties that are robustly useful for... (read 8996 more words →)

[Linkpost] Sam Altman's 2015 Blog Posts Machine Intelligence Parts 1 & 2

Olive Branch

I'm often surprised more people haven't read Open AI CEO Sam Altman's 2015 blog posts Machine Intelligence Part 1 & Part 2. In my opinion, they contain some of the most strong, direct, and clear articulations of why AGI is dangerous from a person at an AGI company.

(Note that the posts were published before OpenAI was founded. There's a helpful wiki of OpenAI history here.)

Hence: a linkpost. I've copied both posts directly below for convenience. I've also bolded a few of the lines I found especially noteworthy.

Machine intelligence, part 1

This is going to be a two-part post—one on why machine intelligence is something we should be afraid of, and one on... (read 2401 more words →)

Christiano (ARC) and GA (Conjecture) Discuss Alignment Cruxes

Andrea_Miotti

Andrea_Miotti, paulfchristiano, Gabriel Alfour, Olive Branch

The following are the summary and transcript of a discussion between Paul Christiano (ARC) and Gabriel Alfour, hereafter GA (Conjecture), which took place on December 11, 2022 on Slack. It was held as part of a series of discussions between Conjecture and people from other organizations in the AGI and alignment field. See our retrospective on the Discussions for more information about the project and the format.

Here's a summary of the discussion, as well as the full transcript below the summary, lightly edited for readability.

Summary

Introduction

GA is pessimistic about alignment being solved because he thinks there is (1) an AGI race to the bottom, (2) alignment is hard in ways that we are... (read 14043 more words →)

Announcing AI Alignment Awards: $100k research contests about goal misgeneralization & corrigibility

Orpheus16

Orpheus16, Olive Branch

We’re grateful to our advisors Nate Soares, John Wentworth, Richard Ngo, Lauro Langosco, and Amy Labenz. We're also grateful to Ajeya Cotra and Thomas Larsen for their feedback on the contests.

TLDR: AI Alignment Awards is running two contests designed to raise awareness about AI alignment research and generate new research proposals. Prior experience with AI safety is not required. Promising submissions will win prizes up to $100,000 (though note that most prizes will be between $1k-$20k; we will only award higher prizes if we receive exceptional submissions.)

You can help us by sharing this post with people who are or might be interested in alignment research (e.g., student mailing lists, FB/Slack/Discord groups.)

What are

... (read 942 more words →)

Ways to buy time

Orpheus16

Orpheus16, Olive Branch, Thomas Larsen

In our last post, we claimed:

On the margin, we think more alignment researchers should work on “buying time” interventions instead of technical alignment research (or whatever else they were doing).

But what does “buying time” actually look like? In this post, we list some interventions that have the potential to buy time (some of which also have other benefits, like increasing coordination, accelerating community growth, and reducing the likelihood that labs deploy dangerous systems).

If you are interested in any of these, please reach out to us. Note also that Thomas has a list of specific technical projects (with details about how they would be implemented), and he is looking for collaborators.

Summary

Ideas to buy time:

Direct

... (read 3415 more words →)

Instead of technical research, more people should focus on buying time

Orpheus16

Orpheus16, Olive Branch, Thomas Larsen

This post is the first in a sequence of posts about AI strategy. In the next post, we’ll provide more examples of “buying time” interventions that we’re excited about.

We’re grateful to Ajeya Cotra, Daniel Kokotajlo, Ashwin Acharya, and Andrea Miotti for feedback on this post.

If anyone is interested in working on "buying time" interventions, feel free to reach out. (Note that Thomas has a list of technical projects with specifics about how to implement them. We also have a list of non-technical projects).

Summary

A few months ago, when we met technical people interested in reducing AI x-risk, we were nearly always encouraging them to try to solve what we see as the core... (read 4170 more words →)

101