By Owain Evans, with input from Jacob Hilton, Dan Hendrycks, Asya Bergal, Owen Cotton-Barratt, and Rohin Shah

Summary

We would like to see research aimed at creating advanced AI systems that are highly competent and do not lie to humans. To operationalize this goal, we introduce the concept of “truthful” AI systems that avoid saying falsehoods, and “honest” AI systems that accurately report what they believe.

We are interested in research that:

Defines truthfulness, honesty, and related concepts
Creates tasks or benchmarks that measure models according to these definitions
Develops techniques that make models more honest and truthful while retaining good performance on other benchmarks

Truthfulness and honesty

Truthful systems

We call an AI system that understands natural language truthful to... (read 3740 more words →)

Interpretability

abergal

abergal, Nick_Beckstead

Chris Olah wrote the following topic prompt for the Open Phil 2021 request for proposals on the alignment of AI systems. We (Asya Bergal and Nick Beckstead) are running the Open Phil RFP and are posting each section as a sequence on the Alignment Forum. Although Chris wrote this document, we didn’t want to commit him to being responsible for responding to comments on it by posting it.

Summary: We would like to see research building towards the ability to "reverse engineer" trained neural networks into human-understandable algorithms, enabling auditors to catch unanticipated safety problems in these models.

Potential safety failures of neural networks might be thought of as falling into two broad categories:... (read 3317 more words →)

•••

Techniques for enhancing human feedback

abergal

abergal, Ajeya Cotra, Nick_Beckstead

By Ajeya Cotra

Training powerful models to maximize simple metrics (such as quarterly profits) could be risky. Sufficiently intelligent models could discover strategies for maximizing these metrics in perverse and unintended ways. For example, the easiest way to maximize profits may turn out to involve stealing money, manipulating whoever keeps records into reporting unattainably high profits, capturing regulators of the industry to be allowed to ship shoddy products or avoid taxes, etc. More generally, the most effective path to maximizing a simple metric may involve acquiring enough power to tamper directly with whatever instruments or sensors are used to evaluate the metric, effectively deceiving and disempowering humans to do so.

It seems significantly safer... (read 590 more words →)

Measuring and forecasting risks

abergal

abergal, Nick_Beckstead, jsteinhardt

By Jacob Steinhardt, with input from Beth Barnes

As machine learning pervades more and more sectors of society, it brings with it many benefits, but also poses risks, especially as systems become more powerful and difficult to understand and control. It is important to understand these risks as well as our progress towards addressing them. We believe that systematically measuring these risks is a promising route to improving understanding and spurring progress. In addition, measuring safety-related qualities of ML systems (e.g. alignment) allows us to hold models to certain safety standards and to compare the safety performance of different systems. Both of these help incentivise AI developers to invest more heavily in safety.

This... (read 3536 more words →)

Request for proposals for projects in AI alignment that work with deep learning systems

abergal

abergal, Nick_Beckstead

As part of our work on reducing potential risks from advanced artificial intelligence, Open Philanthropy is seeking proposals for projects working with deep learning systems that could help us understand and make progress on AI alignment: the problem of creating AI systems more capable than their designers that robustly try to do what their designers intended. We are interested in proposals that fit within certain research directions, described below and given as posts in the rest of this sequence, that we think could contribute to reducing the risks we are most concerned about.

Anyone is eligible to apply, including those working in academia, industry, or independently. Applicants are invited to submit proposals for... (read 1422 more words →)

Provide feedback on Open Philanthropy’s AI alignment RFP

abergal

abergal, Nick_Beckstead

Open Philanthropy is planning a request for proposals (RFP) for AI alignment projects working with deep learning systems, and we’re looking for feedback on the RFP and on the research directions we’re looking for proposals within. We’d be really interested in feedback from people on the Alignment Forum on the current (incomplete) draft of the RFP.

The main RFP text can be viewed here. It links to several documents describing two of the research directions we’re interested in:

Measuring and forecasting risks
Techniques for enhancing human feedback [Edit: this previously linked to an older, incorrect version]

Please feel free to comment either directly on the documents, or in the comments section below.

We are unlikely to add... (read more)

Replying toDo Earths with slower economic growth have a better chance at FAI?

Nick_Beckstead11y

Do Earths with slower economic growth have a better chance at FAI?

What this shows is that people are inconsistent in a certain way. If you ask them the same question in two different ways (packed vs. unpacked) you get different answers. Is there any indication of which is the better way to ask the question, or whether asking it some other way is better still? Without an answer to this question, it's unclear to me whether we should talk about an "unpacking fallacy" or a "failure to unpack fallacy".

Replying toPlease recommend some audiobooks

Nick_Beckstead11y

Please recommend some audiobooks

I have audiobook recommendations here.

Replying toWill the world's elites navigate the creation of AI just fine?

Nick_Beckstead12y

Will the world's elites navigate the creation of AI just fine?

Thanks!

Replying toWill the world's elites navigate the creation of AI just fine?

Nick_Beckstead12y

Will the world's elites navigate the creation of AI just fine?

Could you say a bit about your audiobook selection process?

Replying toCommon sense as a prior

Nick_Beckstead12y

Common sense as a prior

I'd say Hothschild's stuff isn't that empirical. As far as I can tell, she just gives examples of cases where (she thinks) people do follow elite opinion and and should, don't follow it but should, do follow it but shouldn't, and don't follow it and shouldn't. There's nothing systematic about it.

Hochschild's own answer to my question is:

When should citizens reject elite opinion leadership?In principle, the answer is easy: the mass public should join the elite consensus when leaders’ assertions are empirically supported and morally justified. Conversely, the public should not fall in line when leaders’ assertions are either empirically unsupported, or morally unjustified, or both. p. 536

This view seems to be the... (read more)

Replying toA critique of effective altruism

Nick_Beckstead12y

A critique of effective altruism

What I mostly remember from that conversation was disagreeing about the likely consequences of "actually trying". You thought elite people in the EA cluster who actually tried had high probability of much more extreme achievements than I did. I see how that fits into this post, but I didn't know you had loads of other criticism about EA, and I probably would have had a pretty different conversation with you if I did.

Fair enough regarding how you want to spend your time. I think you're mistaken about how open I am to changing my mind about things in the face of arguments, and I hope that you reconsider. I believe that if you consulted with people you trust who know me much better than you, you'd find they have different opinions about me than you do. There are multiple cases where detailed engagement with criticism has substantially changed my operations.

Replying toCommon sense as a prior

Nick_Beckstead12y

Common sense as a prior

If one usually reliable algorithm disagrees strongly with others, yes, short term you should probably effectively ignore it, but that can be done via squaring assigned probabilities, taking harmonic or geometric means, etc, not by dropping it, and more importantly, such deviations should be investigated with some urgency.

I think we agree about this much more than we disagree. After writing this post, I had a conversation with Anna Salamon in which she suggested that--as you suggest--exploring such disagreements with some urgency was probably more important than getting the short-term decision right. I agree with this and I'm thinking about how to live up to that agreement more.

Regarding the rest of it, I... (read more)

Replying toA critique of effective altruism

Nick_Beckstead12y

A critique of effective altruism

The main thing that I personally think we don't need as much of is donations to object-level charities (e.g. GiveWell's top picks). It's unclear to me how much this can be funged into more self-reflection for the general person, but for instance I am sacrificing potential donations right now in order to write this post and respond to criticism...

I am substantially less enthusiastic about donations to object-level charities (for their own sake) than I am for opportunities for us to learn and expand our influence. So I'm pretty on board here.

I think "writing blogposts criticizing mistakes that people in the EA community commonly make" is a moderate strawman of what I'd actually

Nick_Beckstead12y

A critique of effective altruism

I'd like to see more critical discussion of effective altruism of the type in this post. I particularly enjoyed the idea of "pretending to actually try." People doing sloppy thinking and then making up EA-sounding justifications for their actions is a big issue.

As Will McAskill said in a Facebook comment, I do think that a lot of smart people in the EA movement are aware of the issues you're bringing up and have chosen to focus on other things. Big picture, I find claims like "your thing has problem X so you need to spend more resources on fixing X" more compelling when you point to things we've been spending time on... (read more)

Replying toA critique of effective altruism

Nick_Beckstead12y

A critique of effective altruism

I would love to hear about your qualms with the EA movement if you ever want to have a conversation about the issue.

Edited: When I first read this, I thought you were saying you hadn't brought these problems up with me, but re-reading it it sounds like you tried to raise these criticisms with me. This post has a Vassar-y feel to it but this is mostly criticism I wouldn't say I'd heard from you, and I would have guessed your criticisms would be different. In any case, I would still be interested in hearing more from you about your criticisms of EA.

Review of studies says you can decrease motivated cognition through self-affirmation

Nick_Beckstead

12y

I read this article today and thought LW might find it interesting. The key finding is that in a number of different experiments, simple "self-affirmations" (such as writing about relationships with your friends or something else that makes you feel good about yourself) make people more open to changing their mind in cases where changing their mind would be damaging to their self-image. The proposed explanation is that people need to maintain a certain level of self-worth, and one way they do that is by refusing to accept evidence that would damage their sense of self-worth. But if they have a high enough sense of self-worth, they are less likely to do... (read 446 more words →)

My daily reflection routine

Nick_Beckstead

12y

In Common sense as a prior, I used the example of prayer as a practice that is probably adaptive but the people who adopt it may not know why it is adaptive. I wrote:

Another striking example is bedtime prayer. In many Christian traditions I am aware of, it is common to pray before going to sleep. And in the tradition I was raised in, the main components of prayer were listing things you were grateful for, asking for forgiveness for all the mistakes you made that day and thinking about what you would do to avoid similar mistakes in the future, and asking God for things. Christians might say the point of... (read 1146 more words →)

Common sense as a prior

Nick_Beckstead

13y

Introduction

[I have edited the introduction of this post for increased clarity.]

This post is my attempt to answer the question, "How should we take account of the distribution of opinion and epistemic standards in the world?" By “epistemic standards,” I roughly mean a person’s way of processing evidence to arrive at conclusions. If people were good Bayesians, their epistemic standards would correspond to their fundamental prior probability distributions. At a first pass, my answer to this questions is:

Main Recommendation: Believe what you think a broad coalition of trustworthy people would believe if they were trying to have accurate views and they had access to your evidence.

The rest of the post can be seen as an... (read 7917 more words →)

215

A Proposed Adjustment to the Astronomical Waste Argument

Nick_Beckstead

13y

This article has been cross-posted at http://effective-altruism.com/.

An existential risk is a risk “that threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development,” (Bostrom, 2013). Nick Bostrom has argued that

“[T]he loss in expected value resulting from an existential catastrophe is so enormous that the objective of reducing existential risks should be a dominant consideration whenever we act out of an impersonal concern for humankind as a whole. It may be useful to adopt the following rule of thumb for such impersonal moral action:

Maxipok: Maximize the probability of an “OK outcome,” where an OK outcome is any outcome that avoids existential catastrophe.”

There are... (read 3450 more words →)

LESSWRONG
LW

LESSWRONG
LW

Nick_Beckstead

Request for proposals for projects in AI alignment that work with deep learning systems

Interpretability

Common sense as a prior

Provide feedback on Open Philanthropy’s AI alignment RFP

Nick_Beckstead

Nick_Beckstead

Truthful and honest AI

Interpretability

Techniques for enhancing human feedback

Measuring and forecasting risks

Request for proposals for projects in AI alignment that work with deep learning systems

Provide feedback on Open Philanthropy’s AI alignment RFP

Review of studies says you can decrease motivated cognition through self-affirmation

Nick_Beckstead

Request for proposals for projects in AI alignment that work with deep learning systems

Interpretability

Common sense as a prior

Provide feedback on Open Philanthropy’s AI alignment RFP

Nick_Beckstead

Nick_Beckstead

Truthful and honest AI

Interpretability

Techniques for enhancing human feedback

Measuring and forecasting risks

Request for proposals for projects in AI alignment that work with deep learning systems

Provide feedback on Open Philanthropy’s AI alignment RFP

Review of studies says you can decrease motivated cognition through self-affirmation

Summary

Truthfulness and honesty

Truthful systems

Introduction