I'm putting together a list of short and sweet introductions to the dangers of artificial superintelligence.

My target audience is intelligent, broadly philosophical narrative thinkers, who can evaluate arguments well but who don't know a lot of the relevant background or jargon.

My method is to construct a Sequence mix tape — a collection of short and enlightening texts, meant to be read in a specified order. I've chosen them for their persuasive and pedagogical punchiness, and for their flow in the list. I'll also (separately) list somewhat longer or less essential follow-up texts below that are still meant to be accessible to astute visitors and laypeople.

The first half focuses on intelligence, answering 'What is Artificial General Intelligence (AGI)?'. The second half focuses on friendliness, answering 'How can we make AGI safe, and why does it matter?'. Since the topics of some posts aren't obvious from their titles, I've summarized them using questions they address.

 

---

Part I. Building intelligence.

1. Power of Intelligence. Why is intelligence important?

2. Ghosts in the Machine. Is building an intelligence from scratch like talking to a person?

3. Artificial Addition. What can we conclude about the nature of intelligence from the fact that we don't yet understand it?

4. Adaptation-Executers, not Fitness-Maximizers. How do human goals relate to the 'goals' of evolution?

5. The Blue-Minimizing Robot. What are the shortcomings of thinking of things as 'agents', 'intelligences', or 'optimizers' with defined values/goals/preferences?

 

Part II. Intelligence explosion.

6. Optimization and the Singularity. What is optimization? As optimization processes, how do evolution, humans, and self-modifying AGI differ?

7. Efficient Cross-Domain Optimization. What is intelligence?

8. The Design Space of Minds-In-General. What else is universally true of intelligences?

9. Plenty of Room Above Us. Why should we expect self-improving AGI to quickly become superintelligent?

 

Part III. AI risk.

10. The True Prisoner's Dilemma. What kind of jerk would Defect even knowing the other side Cooperated?

11. Basic AI drives. Why are AGIs dangerous even when they're indifferent to us?

12. Anthropomorphic Optimism. Why do we think things we hope happen are likelier?

13. The Hidden Complexity of Wishes. How hard is it to directly program an alien intelligence to enact my values?

14. Magical Categories. How hard is it to program an alien intelligence to reconstruct my values from observed patterns?

15. The AI Problem, with Solutions. How hard is it to give AGI predictable values of any sort? More generally, why does AGI risk matter so much?

 

Part IV. Ends.

16. Could Anything Be Right? What do we mean by 'good', or 'valuable', or 'moral'?

17. Morality as Fixed Computation. Is it enough to have an AGI improve the fit between my preferences and the world?

18. Serious Stories. What would a true utopia be like?

19. Value is Fragile. If we just sit back and let the universe do its thing, will it still produce value? If we don't take charge of our future, won't it still turn out interesting and beautiful on some deeper level?

20. The Gift We Give To Tomorrow. In explaining value, are we explaining it away? Are we making our goals less important?

 

Summary: Five theses, two lemmas, and a couple of strategic implications.

---

 

All of the above were written by Eliezer Yudkowsky, with the exception of The Blue-Minimizing Robot (by Yvain), Plenty of Room Above Us and The AI Problem (by Luke Muehlhauser), and Basic AI Drives (a wiki collaboration). Seeking a powerful conclusion, I ended up making a compromise between Eliezer's original The Gift We Give To Tomorrow and Raymond Arnold's Solstice Ritual Book version. It's on the wiki, so you can further improve it with edits.

 

Further reading:

• Three Worlds Collide (Normal), by Eliezer Yudkowsky
  • a short story vividly illustrating how alien values can evolve.
• So You Want to Save the World, by Luke Muehlhauser
  • an introduction to the open problems in Friendly Artificial Intelligence.
• Intelligence Explosion FAQ, by Luke Muehlhauser
  • a broad overview of likely misconceptions about AI risk.
• The Singularity: A Philosophical Analysis, by David Chalmers
  • a detailed but non-technical argument for expecting intelligence explosion, with an assessment of the moral significance of synthetic human and non-human intelligence.

 

I'm posting this to get more feedback for improving it, to isolate topics for which we don't yet have high-quality, non-technical stand-alone introductions, and to reintroduce LessWrongers to exceptionally useful posts I haven't seen sufficiently discussed, linked, or upvoted. I'd especially like feedback on how the list I provided flows as a unit, and what inferential gaps it fails to address. My goals are:

A. Via lucid and anti-anthropomorphic vignettes, to explain AGI in a way that encourages clear thought.

B. Via the Five Theses, to demonstrate the importance of Friendly AI research.

C. Via down-to-earth meta-ethics, humanistic poetry, and pragmatic strategizing, to combat any nihilisms, relativisms, and defeatisms that might be triggered by recognizing the possibility (or probability) of Unfriendly AI.

D. Via an accessible, substantive, entertaining presentation, to introduce the raison d'être of LessWrong to sophisticated newcomers in a way that encourages further engagement with LessWrong's community and/or content.

What do you think? What would you add, remove, or alter?

One of the things I loved about studying liberal arts is that you actually got to know your professors. They would discuss their personal experiences in a topic ("Here's what I did during the feminist movement.."), you might get slide shows from their vacation in the country of study, or even invited to their house for a group dinner. 

Going into engineering was rather jarring for me in that regard. The vast majority of professors would come to class, lecture on the topic, and that would be it. They might share what their specific field of study was, but they rarely shared any personal details. It actually made it harder for me to learn, because it was like "Who is this person who is talking to me?"

(I think a large part of this for me personally was because I am motivated by a desire to please, and so if I liked my professors, then I wouldn't want to inconvenience them by handing things in late, or bore them by giving them another sub-par paper to read. But that's another discussion...)

I've noticed that Less Wrong is similar in some ways. We may know about each other's views on particular topics, and general fields of study, but we know very little about each other as people, unless a personal topic happens to be related to a particular rationalist study. Even the intro thread set up here focuses mainly on non-personal information.

For example, a Generic Intro post right now would be something like: "I'm X years old. From place Y. The fields I study/want to study are Z. Here's what college/HS was/is like for me. I have akrasia." Pretty boring, right? INSTEAD, the things I would be interested in knowing about my fellow LWers include: "On my time off I enjoy underwater basketweaving and climbing Mt Kilamanjaro. I have 6 young daughters and a dog named Grrr. I love pesto. etc"

From a rational perspective, an argument could be made that it's easier to have constructive arguments that remain civil when you humanize the people you are speaking with. 

 

---

 

I was wondering how other LWers feel on the subject. Do you like that our discussions are un-hampered by personal data? Do you like the idea of providing personal intros? Do you not want to provide personalish information for safety reasons, or because you don't think it's anyone business?

If you think you might need help writing a personal intro, I wrote [a general guide](http://lesswrong.com/lw/8nq/more_personal_introductions/5d4e) on the topic in the comments below. 

Note: I predict there will be two types of response to this post. People discussing how they feel about this (Meta-Comments), and people giving personal introductions (Intros). To make navigating the responses easier, I am trying an experiment where I set up a meta-comment thread and a personal introduction thread. 

PLEASE PLACE COMMENTS ABOUT THIS IDEA IN META-COMMENT THREAD, AND COMMENTS INTRODUCING YOURSELF IN INTRO THREAD.

 

Edited to make it more clear to focus on personality, hobbies, likes/dislikes, and NOT on what you study, or school.
ETA- Added link to "How to Write Personal Intro" comment