Comment Permalink

Ben Pringle7y60

I saw a talk earlier this year that mentioned this 2015 Corrigibility paper as a good starting point for someone new to alignment research. If that's still true, I started writing up some thoughts on a possible generalization of the method in that paper.

Anyway, submitting this draft early to hopefully get some feedback whether I'm on the right track:

GeneralizedUtilityIndifference_Draft_Latest.pdf (edited)

The new version does better on sub-agent shutdown and eliminates the "managing the news" problem.

(Let me know if someone already thought of this approach!)

EDIT 2017-11-09: filled in the section on the $n$ -action model.

cousin_it7y20

Ben, thank you! Sent you an email.

See in context

95 Announcing the AI Alignment Prize

by cousin_it

3rd Nov 2017

2 min read

95

Stronger than human artificial intelligence would be dangerous to humanity. It is vital any such intelligence’s goals are aligned with humanity's goals. Maximizing the chance that this happens is a difficult, important and under-studied problem.

To encourage more and better work on this important problem, we (Zvi Mowshowitz and Vladimir Slepnev) are announcing a $5000 prize for publicly posted work advancing understanding of AI alignment, funded by Paul Christiano.

This prize will be awarded based on entries gathered over the next two months. If the prize is successful, we will award further prizes in the future.

The prize is not backed by or affiliated with any organization.

Rules

Your entry must be published online for the first time between November 3 and December 31, 2017, and contain novel ideas about AI alignment. Entries have no minimum or maximum size. Important ideas can be short!

Your entry must be written by you, and submitted before 9pm Pacific Time on December 31, 2017. Submit your entries either as links in the comments to this post, or by email to apply@ai-alignment.com. We may provide feedback on early entries to allow improvement.

We will award $5000 to between one and five winners. The first place winner will get at least $2500. The second place winner will get at least $1000. Other winners will get at least $500.

Entries will be judged subjectively. Final judgment will be by Paul Christiano. Prizes will be awarded on or before January 15, 2018.

What kind of work are we looking for?

AI Alignment focuses on ways to ensure that future smarter than human intelligence will have goals aligned with the goals of humanity. Many approaches to AI Alignment deserve attention. This includes technical and philosophical topics, as well as strategic research about related social, economic or political issues. A non-exhaustive list of technical and other topics can be found here.

We are not interested in research dealing with the dangers of existing machine learning systems commonly called AI that do not have smarter than human intelligence. These concerns are also understudied, but are not the subject of this prize except in the context of future smarter than human intelligence. We are also not interested in general AI research. We care about AI alignment, which may or may not also advance the cause of general AI research.

(Addendum: the results of the prize and the rules for the next round have now been announced.)

2017-2019 AI Alignment PrizeBounties (closed)AI

Frontpage

95

Mentioned in

1902018 AI Alignment Literature Review and Charity Comparison

103Money Stuff

81Announcement: AI alignment prize winners and next round

39Beware of black boxes in AI alignment research

38What would you need to be motivated to answer "hard" LW questions?

Announcing the AI Alignment Prize

3Adrià Garriga-alonso

New Comment

78 comments, sorted by

top scoring

Click to highlight new comments since: Today at 12:14 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Scott Garrabrant7y60

Here is my submission.

Thank you for motivating me to write this blog post I have been putting off for a while.

Disclaimer: If you want to only measure the contribution that came November or later, compare to this post, which has one fewer category, no names, fewer examples, nothing about mitigation, and worse presentation.

I think this is an important idea, so I appreciate feedback, especially about presentation.

4cousin_it7y

Acknowledged, and thank you! I think that's a great post. If you want to make it better, the easiest way is by adding simple real world examples for each section.

1Scott Garrabrant7y

Another Disclaimer: The outline at the beginning was added after the deadline, thanks to Raemon and other people who provided examples.

[-]Caspar Oesterheld7y60

You don't mention decision theory in your list of topics, but I guess it doesn't hurt to try.

I have thought a bit about what one might call the "implementation problem of decision theory". Let's say you believe that some theory of rational decision making, e.g., evidential or updateless decision theory, is the right one for an AI to use. How would you design an AI to behave in accordance with such a normative theory? Conversely, if you just go ahead and build a system in some existing framework, how would that AI behave in Newcomb-... (read more)

3cousin_it7y

Caspar, thanks for the amazing entry! Acknowledged.

[-]Ben Pringle7y60

Anyway, submitting this draft early to hopefully get some feedback whether I'm on the right track:

GeneralizedUtilityIndifference_Draft_Latest.pdf (edited)

The new version does better on sub-agent shutdown and eliminates the "managing the news" problem.

(Let me know if someone already thoug... (read more)

2cousin_it7y

Ben, thank you! Sent you an email.

[-]Stuart_Armstrong7y60

Should I submit? Working on this is my job, so it's maybe better to encourage others to come on board?

5cousin_it7y

We certainly don't want to exclude experts! Please do submit.

[-]Zvi7y60

What should we be doing to help get more people to enter, whether by spreading the word or another way? We want this to work and result in good things, and it's iteration one so doubtless a lot we're not doing right.

8whpearson7y

Some random initial thoughts. Post on the SSC open thread ? Or the EA forum open thread (maybe the EA subreddit too). I've seen it posted to the control problem reddit. I'll post it on the ai danmark safety facebook page, although I've never managed to go to one of their reading groups (it is now pending). Ask nicely the people running lesserwrong to see if you can see the referrer for where traffic comes in to this thread, this will give you an idea where most of the traffic comes from. To get more people to enter, imagine you were running the competition previously, pick N articles out there on the internet and link them as things that would be short listed. This would give people an idea of what you are looking for. Try and pick a diverse range else you might get articles in a cluster. Perhaps think about trying to get some publicity to sweeten the deal, e.g. the winner also gets featured in X prestigious place (if the submitter wants it to be). Although maybe only after the quality has been shown to be high enough, after the first couple of iterations.

1habryka7y

Happy to give you any analytics data for the page.

2cousin_it7y

Can you send them to Zvi and me? (vladimir.slepnev at gmail, thezvi at gmail)

6Raemon7y

Yeah, I had an initial gut sense of "oh man this seems important and but I'm worried it'd quietly fade out of consciousness by default." Much of my advice would be whpearson's. Some additional thoughts (I think mostly fleshing out why I think whpearson's suggestions are important) i. Big Activation Costs You are asking people to do a hard thing. You'd providing money to incentivize them, but people are lazy - they will forget, or start doing it but not get around to finish or not get around to finishing until too late. Anything to reduce the activation cost is good. 1) Maybe have the first thing you ask is for people to apply if they might be interested, with as low a cost to doing so as possible (while gaining at least some information about people and weeding out dead-wood). This gets people slightly committed, and gives you the opportunity to spam a much narrower subset of people to remind them. (see spam section) 2) It's ambiguous to me what kind of writing you're looking for, which in turn makes me unsure if it's be a good use of my time to work on this, which makes me hesitate. (I'm currently assuming that this is not the right use of my talents both for altruistic and selfish reasons, but I can imagine a slightly different version of me for whom it'd be ambiguous) Whpearson's "list good existing articles, as diverse as possible" helps counteract part of this, but still doesn't answer questions like "should I be doing this if this is currently my day job? Presumably the point is to get more people workin on this." (and the correlary: if professional AI safety workers are submitting, what chance do I have of contributing something useful?) (Relatedly - I'd originally thought you should spell out what sort of questions you were looking to resolve, then saw you had linked to Paul Christiano's doc. I think attempting to summarize the doc might accidentally focus on too narrow a domain, but the current linking of the doc is so small I missed it the first ti

2avturchin7y

Put it in all relevant facebook groups?

1habryka7y

I think this is the most important thing, and I would be happy to help with that.

2cousin_it7y

That would be great! Do you need any input from us to do it?

[-]Dr_Manhattan7y60

Zvi/Vladimir, what's your role in this - are you the judges?

6cousin_it7y

Organizing and judging. Might bring in other judges if things go well.

[-]Dan Fitch7y50

I don't know if this is a useful "soft" submission, considering I am still reading and learning in the area.

But I think the current metaphors (paperclips, etc.) are not very persuasive for convincing folks in the world at large that value alignment is a BIG, HARD PROBLEM. Here is my attempt to add a possibly-new metaphor to the mix: https://nilscript.wordpress.com/2017/11/26/parenting-alignment-problem/

2cousin_it7y

Thank you! Acknowledged. Do you have an email address for contact?

[-]Shmi7y40

Posted on my blog, but might as well link it here. Not of the quality that Paul Christiano seeks, but might be of some interest, though many of the same point points have been discussed over and over here and elsewhere before.

3Defective Altruist7y

Your link is broken, did you mean to link https://edgeofgravity.wordpress.com/2017/12/31/ai-alignment-bubblicious/ ?

2cousin_it7y

Thank you! Acknowledged (though your link didn't work for me, Defective Altruist's did). Do you have an email address for contact?

2Shmi7y

shminux at gmail should work. Thank you for the acknowledgment! Tried to fix the link above, not sure how well it worked.

[-]Defective Altruist7y40

Submission: http://futilitymonster.tumblr.com/post/169068920534/the-true-ai-box

Contact: defectivealtruist at g mail

2cousin_it7y

Thank you! Acknowledged.

[-]Daniel Wallis7y40

Is "publishing" on google docs ok? Here's a link:

https://docs.google.com/document/d/1hIzJNNDfWKwAK-lSgs_w-CYa15b0SjdRNDzmh5jJMMU/edit?usp=sharing

2cousin_it7y

Thank you! Acknowledged. Do you have an email address for contact?

1Daniel Wallis7y

daniel.wallis.9000@gmail.com

[-]Joseph Shipman7y40

OK, I went on a rant and revived my blog after 4 years of inactivity because entries aren't supposed to be entered as comments but are supposed to be linked to instead.

https://polymathblogger.wordpress.com/2017/11/12/acknowledge-the-elephant-entry-for-ai-alignment-prize/

2cousin_it7y

Thanks! Can you give your email address so we can send feedback?

1Joseph Shipman7y

Just comment on the blog or here or both, if you want to send private feedback try JoeShipman a-with-a-circle-around-it aol end-of-sentence-punctuation com

[-]Sergej Xarkonnen7y40

my idea: https://docs.google.com/document/d/e/2PACX-1vQ3131oaC2JhxafeR77x3nbuOcPRoxLFI0PQvxcYt6N8IqK-FFV6mcK3CMXeEpZlTxjSmSXpvYYbbq7/pub

2cousin_it7y

Thanks! Sent you feedback by email.

[-]jimrandomh7y40

Are there any limitations on number of submissions per person (where each submission is a distinct idea)? On number of wins per person?

3cousin_it7y

One win per person, and it's okay to have many ideas. Might be more convenient if you submit them as one package though.

[-]John_Maxwell7y30

Here's my entry: Friendly AI through Ontology Autogeneration. Am I allowed to keep making improvements to it even after the deadline has passed? (Doing so at my own risk, i.e. if it so happens that you've already read & judged my essay before I make my improvements, and my improvements aren't going to affect my chances of winning, that's my problem.)

2cousin_it7y

Can you make a snapshot frozen at the moment of deadline and give us a URL to it? That would be the most fair decision for the other contestants, I think.

2John_Maxwell7y

OK, I won't make further modifications to the version at the URL in my comment above. EDIT: Now that the judging is over, I am making some modifications, but nothing major.

[-]Patterns_Everywhere7y30

Here's my entry. I think it's what you want... Hosted on DocDroid.

http://docdro.id/bUVo61P

1Patterns_Everywhere7y

I was going to add another section to the above report with diagrams and explanations but I wouldn't get to finish it like I wanted to in time. But if you want the basic diagram with no explanations to understand it better I just uploaded the basic flowchart. http://docdro.id/hK8OpYJ Just apply the document sections to the parts.

3cousin_it7y

Thank you! Acknowledged. Do you have an email address for contact?

1Patterns_Everywhere7y

Just sent an Email to the contest Email listed at the top. I assume that is fine. Happy New Years Everyone!

[-]Jeremy Popejoy7y30

Hello :) I’ve created this as a framework for guiding our future with AI http://peridotai.com/call-to-artists/ AND to bootstrap interest in my art and thoughts here at https://quantumsymbol.com

2cousin_it7y

Thank you! Acknowledged.

[-]Joseph Shipman7y30

You should think about the incentives of posting early in the 2 month window rather than late. Later entries will be influenced by earlier entries so you have a misalignment between wanting to win the prize and wanting to advance the conversation sooner. Christiano ought to announce that if one entry builds in a valuable way on an earlier entry by someone else, the earlier submitter will also gain subjective judgy-points in a way that he, Paul, affirms is calibrated neither to penalize early entry nor to discourage work that builds on earlier entries.

[-]Adrià Garriga-alonso7y30

Is it possible to enter the contest as a group? Meaning, can the article written for the contest have several coauthors?

2cousin_it7y

Yes!

2whpearson7y

Jacob Edelman already asked this question and got an affirmative.

[-]James D Miller7y30

Are you looking for entries with actionable information, or would you be interested in a paper showing, for example, that AI alignment might not be as big a problem as we thought but not for a reason that will help us solve the AI alignment problem?

4cousin_it7y

Yes, a paper like that could qualify.

[-]whpearson7y30

Should've saved my decsion alignment loop post a few days. Maybe an expansion of it? Hmm.

4cousin_it7y

Yes, an expansion of that post would qualify.

2whpearson7y

How much should I try to make it self-contained?

2cousin_it7y

I'd prefer a self-contained thing. In the extreme case (which might not apply to you), an entry with many links to the author's previous writings might be hard to judge unless these writings are already well known.

2whpearson7y

Here is a draft

[-]ExCeph7y20

Submitting this entry for your consideration: https://www.lesserwrong.com/posts/bkoeQLTBbodpqHePd/ai-goal-alignment-entry-how-to-teach-a-computer-to-love. I'll email it as well. Your commitment to this call for ideas is much appreciated!

[-]Berick Cook7y20

My submission is on my project blog: https://airis-ai.com/2017/12/31/friendly-ai-via-agency-sustainment/

Thank you for hosting this excellent competition! It was very inspiring. This is an idea I've been bouncing around in the back of my mind for several months now, and it is your competition that prompted me to refine it, flesh it out, and put it to paper.

My contact email is berickcook@gmail.com

3cousin_it7y

Acknowledged, thank you!

[-]interstice7y20

Submission:

https://www.lesserwrong.com/posts/ytph8t6AcxPcmJtDh/formal-models-of-complexity-and-evolution

2cousin_it7y

Received, thank you! Can you give a contact email address?

1interstice7y

usernameneeded@gmail.com Hope it's not too late, but I also meant for this post(linked in original) to be part of my entry: https://www.lesserwrong.com/posts/ra4yAMf8NJSzR9syB/a-candidate-complexity-measure

[-]Roland Pihlakas7y20

Hello! My newest proposal:

https://medium.com/threelaws/making-ai-less-dangerous-2742e29797bd

I would like to propose a certain kind of AI goal structures that would be an alternative to utility maximisation based goal structures. The proposed alternative framework would make AI significantly safer, though it would not guarantee total safety. It can be used at strong AI level and also much below, so it is well scalable. The main idea would be to replace utility maximisation with the concept of homeostasis.

3cousin_it7y

Acknowledged, thank you!

[-]Glen Slade7y20

Hi all. I have posted my competition entry on my blog here:

https://glenslade.wordpress.com/2017/12/31/the-role-of-learning-and-responsibility-in-ai-alignment/

Happy holidays!

2cousin_it7y

Thank you for the entry! Do you have an email address for contact?

1Glen Slade7y

Hi cousin_it I emailed a pdf to the competition address so hopefully you can access my email there. If not, please let me know the best way to send to you without posting itpublically. Thanks

[-]alexsalt7y20

My entry:

Raising Moral AI

Is it easier to teach a robot to stay safe by not tearing off its own limbs and not drilling holes in its head and not touching lava and not falling from a cliff and so on ad infinitum, or introduce pain as inescapable consequence of such actions and let robot experiment and learn?

Similarly, while trying to create a safe AGI, it is futile to make exhaustive and non-contradictory set of rules (values, policies, laws, committees) due to infinite complexity. A powerful AGI agent might find an exception or conflict in rules and... (read more)

0Justus Eapen7y

I don't see why this has been downvoted so many times. It is likely to be the only way of ensuring the value-alignment we seek. It is based on ancient wisdom (cut the trees that bear bad fruit) and prioritizes safety by cordoning off AGI agents.

[-]Jacob Edelman7y20

Are teams allowed to make submissions?

2cousin_it7y

Yes.

[-]avturchin7y20

I have unpublished text on the topic and will put a draft online in the next couple of weeks, and will apply it to the competition. I will add URL here when it will be ready.

4cousin_it7y

Good. If you submit it early we'll give feedback so you can improve it before the deadline if needed.

1avturchin7y

https://www.lesserwrong.com/posts/CDWsjQr8KDuj69fTJ/message-to-any-future-ai-there-are-several-instrumental This is my early submission and I hope to get useful feedback.

1avturchin7y

Thanks! I will reread it and submit soon.

[-]alexsalt7y10

Any winners?

1Gyrodiot7y

Winners have just been announced here.

Moderation Log