Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

How can I reduce existential risk from AI?

44 Post author: lukeprog 13 November 2012 09:56PM

Suppose you think that reducing the risk of human extinction is the highest-value thing you can do. Or maybe you want to reduce "x-risk" because you're already a comfortable First-Worlder like me and so you might as well do something epic and cool, or because you like the community of people who are doing it already, or whatever.

Suppose also that you think AI is the most pressing x-risk, because (1) mitigating AI risk could mitigate all other existential risks, but not vice-versa, and because (2) AI is plausibly the first existential risk that will occur.

In that case, what should you do? How can you reduce AI x-risk?

It's complicated, but I get this question a lot, so let me try to provide some kind of answer.

 

Meta-work, strategy work, and direct work

When you're facing a problem and you don't know what to do about it, there are two things you can do:

1. Meta-work: Amass wealth and other resources. Build your community. Make yourself stronger. Meta-work of this sort will be useful regardless of which "direct work" interventions turn out to be useful for tackling the problem you face. Meta-work also empowers you to do strategic work.

2. Strategy work: Purchase a better strategic understanding of the problem you're facing, so you can see more clearly what should be done. Usually, this will consist of getting smart and self-critical people to honestly assess the strategic situation, build models, make predictions about the effects of different possible interventions, and so on. If done well, these analyses can shed light on which kinds of "direct work" will help you deal with the problem you're trying to solve.

When you have enough strategic insight to have discovered some interventions that you're confident will help you tackle the problem you're facing, then you can also engage in:

3. Direct work: Directly attack the problem you're facing, whether this involves technical research, political action, particular kinds of technological development, or something else.

Thinking with these categories can be useful even though the lines between them are fuzzy. For example, you might have to do some basic awareness-raising in order to amass funds for your cause, and then once you've spent those funds on strategy work, your strategy work might tell you that a specific form of awareness-raising is useful for political action that counts as "direct work." Also, some forms of strategy work can feel like direct work, depending on the type of problem you're tackling.


Meta-work for AI x-risk reduction

Make money. Become stronger. Build a community, an audience, a movement. Store your accumulated resources in yourself, in your community, in a donor-advised fund, or in an organization that can advance your causes better than you can as an individual.

  1. Make money: In the past 10 years, many people have chosen to start businesses or careers that (1) will predictably generate significant wealth they can spend on AI x-risk reduction, (2) will be enjoyable enough to "stick with it," and (3) will not create large negative externalities. But certainly, the AI x-risk reduction community needs a lot more people to do this! If you want advice, the folks at 80,000 Hours are the experts on "ethical careers" of this sort.

  2. Become stronger: Sometimes it makes sense to focus on improving your productivity, your research skills, your writing skills, your social skills, etc. before you begin using those skills to achieve your goals. Example: Vladimir Nesov has done some original research, but mostly he has spent the last few years improving his math skills before diving into original research full-time.

  3. Build a community / a movement: Individuals can change the world, but communities and movements can do even better, if they're well-coordinated. Read What Psychology Can Teach Us About Spreading Social Change. Launch (or improve) a Less Wrong group. Join a THINK group. Help grow and improve the existing online communities that tend to have high rates of interest in x-risk reduction: LessWrong, Singularity Volunteers, and 80,000 Hours. Help write short primers on crucial topics. To reach a different (and perhaps wealthier, more influential) audience, maybe do help with something like the Singularity Summit.

  4. Develop related skills in humanity. In other words, "make humanity stronger in ways that are almost certainly helpful for reducing AI x-risk" (though strategic research may reveal they are not nearly the most helpful ways to reduce AI x-risk). This might include, for example, getting better at risk analysis with regard to other catastrophic risks, or improving our generalized forecasting abilities by making wider use of prediction markets.

  5. Fund a person or organization doing (3) or (4) above. The Singularity Institute probably does more AI x-risk movement building than anyone, followed by the Future of Humanity Institute. There are lots of organizations doing things that plausibly fall under (4).

Note that if you mostly contribute to meta work, you want to also donate a small sum (say, $15/mo) to strategy work or direct work. If you only contribute to meta work for a while, an outside view (around SI, anyway) suggests there's a good chance you'll never manage to ever do anything non-meta. A perfect Bayesian agent might not optimize this way, but optimal philanthropy for human beings works differently.

 

Strategy work for AI x-risk reduction

How can we improve our ability to do long-term technological forecasting? Is AGI more likely to be safe if developed sooner (Goertzel & Pitt 2012) or later (Muehlhauser & Salamon 2012)? How likely is hard takeoff vs. soft takeoff? Could we use caged AGIs or WBEs to develop safe AGIs or WBEs? How might we reduce the chances of an AGI arms race (Shulman 2009)? Which interventions should we prioritize now, to reduce AI x-risk?

These questions and many others have received scant written analysis — unless you count the kind of written analysis that is (1) written with much vagueness and ambiguity, (2) written in the author's own idiosyncratic vocabulary, (3) written with few citations to related work, and is (4) spread across a variety of non-linear blog articles, forum messages, and mailing list postings. (The trouble with that kind of written analysis is that it is mostly impenetrable or undiscoverable to most researchers, especially the ones who are very busy because they are highly productive and don't have time to comb through 1,000 messy blog posts.)

Here, then, is how you might help with strategy work for AI x-risk reduction:

  1. Consolidate and clarify the strategy work currently only available in a disorganized, idiosyncratic form. This makes it easier for researchers around the world to understand the current state of play, and build on it. Examples include Chalmers (2010), Muehlhauser & Helm (2012), Muehlhauser & Salamon (2012), Yampolskiy & Fox (2012), and (much of) Nick Bostrom's forthcoming scholarly monograph on machine superintelligence.

  2. Write new strategic analyses. Examples include Yudkowsky (2008), Sotala & Valpola (2012), Shulman & Sandberg (2010), Shulman (2010), Shulman & Armstrong (2009), Bostrom (2012), Bostrom (2003), Omohundro (2008), Goertzel & Pitt (2012), Yampolskiy (2012), and some Less Wrong posts: Muehlhauser (2012), Yudkowsky (2012), etc. See here for a list of desired strategic analyses (among other desired articles).

  3. Assist with (1) or (2), above. This is what SI's "remote researchers" tend to do, along with many SI volunteers. Often, there are "chunks" of research that can be broken off and handed to people who are not an article's core authors, e.g. "Please track down many examples of the 'wrong wish' trope so I can use a vivid example in my paper" or "Please review and summarize the part of the machine ethics literature that has to do with learning preferences from examples."

  4. Provide resources and platforms that make it easier for researchers to contribute to strategy work. Things like my AI risk bibliography and list of forthcoming and desired articles on AI risk make it easier for researchers to find relevant work, and to know what projects would be helpful to take on. SI's public BibTeX file and Mendeley group make it easier for researchers to find relevant papers. The AGI conference, and volumes like Singularity Hypotheses, provide publishing venues for researchers in this fledgling field. Recent improvements to the Less Wrong wiki will hopefully make it easier for researchers to understand the (relatively new) concepts relevant to AI x-risk strategy work. A scholarly AI risk wiki would be even better. It would also help to find editors of prestigious journals who are open to publishing well-written AGI risk papers, so that university researchers can publish on these topics without hurting their chances to get tenure.

  5. Fund a person or organization doing any of the above. Again, the most obvious choices are the Singularity Institute or the Future of Humanity Institute. Most of the articles and "resources" above were produced by either SI or FHI. SI offers more opportunities for (3). The AGI conference is organized by Ben Goertzel and others, who are of course always looking for sponsors for the AGI conference.


Direct work for AI x-risk reduction

We are still at an early stage in doing strategy work on AI x-risk reduction. Because of this, most researchers in the field feel pretty uncertain about which interventions would be most helpful for reducing AI x-risk. Thus, they focus on strategic research, so they can purchase more confidence about which interventions would be helpful.

Despite this uncertainty, I'll list some interventions that at least some people have proposed for mitigating AI x-risk, focusing on the interventions that are actionable today.

  • Safe AGI research? Many proposals have been made for developing AGI designs with internal motivations beneficial to humans — including Friendly AI (Yudkowsky 2008) and GOLEM (Goertzel 2010) — but researchers disagree about which approaches are most promising (Muehlhauser & Helm 2012; Goertzel & Pitt 2012).

  • AI boxing research? Many proposals have been made for confining AGIs (Yampolskiy 2012; Armstrong et al. 2012). But such research programs may end up being fruitless, since it may be that a superintelligence will always be able to think its way out of any confinement designed by a human-level intelligence.

  • AI safety promotion? One may write about the importance of AI safety, persuade AI safety researchers to take up a greater concern for safety, and so on.

  • Regulate AGI development? Or not? Hughes (2001) and Daley (2011) call for regulation of AGI development. To bring this about, citizens could petition their governments and try to persuade decision-makers. McGinnis (2010) and Goertzel & Pitt (2012), however, oppose AGI regulation.

  • Accelerate AGI? Or not? Muehlhauser & Salamon (2012) recommend accelerating AGI safety research relative to AGI capabilities research, so that the first AGIs have a better chance of being designed to be safe. Goertzel & Pitt (2012), in contrast, argue that "the pace of AGI progress is sufficiently slow that practical work towards human-level AGI is in no danger of outpacing associated ethical theorizing," and argue that AGI development will be safest if it happens sooner rather than later.

  • Accelerate WBE? Or not? Participants in a 2011 workshop concluded that accelerating WBE probably increases AI x-risk, but Koene (2012) argues that WBE is safer than trying to create safe AGI.

  • Ban AGI and hardware development? Or not? Joy (2000) famously advocated a strategy of relinquishment, and Berglas (2009) goes so far as to suggest we abandon further computing power development. Most people, of course, disagree. In any case, it is doubtful that groups with such views could overcome the economic and military incentives for further computing and AGI development.

  • Foster positive values? Kurzweil (2005) and others argue that one way to increase the odds that AGIs will behave ethically is to increase the chances that the particular humans who create them are moral. Thus, one might reduce AI x-risk by developing training and technology for moral enhancement (Persson and Savulescu 2008).

  • Cognitive enhancement? Some have wondered whether the problem of safe AGI is so difficult that it will require cognitive enhancement for humans to solve it. Meanwhile, others worry that cognitive enhancement will only accelerate the development of dangerous technologies (Persson and Savulescu 2008)

Besides engaging in these interventions directly, one may of course help to fund them. I don't currently know of a group pushing for AGI development regulations, or for banning AGI development. You could accelerate AGI by investing in AGI-related companies, or you could accelerate AGI safety research (and AI boxing research) relative to AGI capabilities research by funding SI or FHI, who also probably do the most AI safety promotion work. You could fund research on moral enhancement or cognitive enhancement by offering grants for such research. Or, if you think "low-tech" cognitive enhancement is promising, you could fund organizations like Lumosity (brain training) or the Center for Applied Rationality (rationality training).

 

Conclusion

This is a brief guide to what you can do to reduce existential risk from AI. A longer guide could describe the available interventions in more detail, and present the arguments for and against each one. But that is "strategic work," and requires lots of time (and therefore money) to produce.

My thanks to Michael Curzi for inspiring this post.

Comments (79)

Comment author: Alex_Altair 11 November 2012 11:32:28PM 15 points [-]

Thanks for putting all this stuff in one place!

It makes me kind of sad that we still have more or less no answer to so many big, important questions. Does anyone else share this worry?

Comment author: lukeprog 12 November 2012 01:10:00AM 8 points [-]

Notice also that until now, we didn't even have a summary of the kind in this post! So yeah, we're still at an early stage of strategic work, which is why SI and FHI are spending so much time on strategic work.

I'll note, however, that I expect significant strategic insights to come from the technical work (e.g. FAI math). Such work should will give us insight into how hard the problems actually are, what architectures look most promising, to what degree the technical work can be outsourced to the mainstream academic community, and so on.

Comment author: Alex_Altair 12 November 2012 06:54:16AM 14 points [-]

I expect significant strategic insights to come from the technical work (e.g. FAI math).

Interesting point. I'm worried that, while FAI math will help us understand what is dangerous or outsourceable from our particular path, many many other paths to AGI are possible, and we won't learn from FAI math which of those other paths are dangerous or likely.

I feel like one clear winning strategy is safety promotion. It seems that almost no bad can come from promoting safety ideas among AI researchers and investors. It also seems relatively easy, in that requires only regular human skills of networking, persuasion, et cetera.

Comment author: roystgnr 13 November 2012 11:35:58PM 8 points [-]

You're probably right about safety promotion, but calling it "clear" may be an overstatement. A possible counterargument:

Existing AI researchers are likely predisposed to think that their AGI is likely to naturally be both safe and powerful. If they are exposed to arguments that it will instead naturally be both dangerous and very powerful (the latter half of the argument can't be easily omitted; the potential danger is in part because of the high potential power), would it not be a natural result of confirmation bias for the preconception-contradicting "dangerous" half of the argument to be disbelieved and the preconception-confirming "very powerful" half of the argument to be believed?

Half of the AI researcher interviews posted to LessWrong appear to be with people who believe that "Garbage In, Garbage Out" only applies to arithmetic, not to morality. If the end result of persuasion is that as many as half of them have that mistake corrected while the remainder are merely convinced that they should work even harder, that may not be a net win.

Comment author: danieldewey 16 November 2012 09:48:06AM 3 points [-]

believe that "Garbage In, Garbage Out" only applies to arithmetic, not to morality

Catchy! Mind if I steal a derivative of this?

Comment author: roystgnr 19 November 2012 11:22:31PM 7 points [-]

I've lost all disrespect for the "stealing" of generic ideas, and roughly 25% of the intended purpose of my personal quotes files is so that I can "rob everyone blind" if I ever try writing fiction again. Any aphorisms I come up with myself are free to be folded, spindled, and mutilated. I try to cite originators when format and poor memory permit, and receiving the same favor would be nice, but I certainly wouldn't mind seeing my ideas spread completely unattributed either.

Comment author: [deleted] 20 November 2012 06:29:43PM 0 points [-]

I've lost all disrespect for the "stealing" of generic ideas

Relevant TED talk

Comment author: danieldewey 20 November 2012 11:58:01AM 0 points [-]

Noted; thanks.

Comment author: Alex_Altair 14 November 2012 12:01:38AM 1 point [-]

Yeah, quite possibly. But I wouldn't want people to run into analysis paralysis; I still think safety promotion is very likely to be a great way to reduce x-risk.

Comment author: [deleted] 20 November 2012 06:38:18PM *  0 points [-]

Half of the AI researcher interviews posted to LessWrong appear to be with people who believe that "Garbage In, Garbage Out" only applies to arithmetic, not to morality.

Does 'garbage in, garbage out' apply to morality, or not?

Comment author: MugaSofer 16 November 2012 01:39:15PM 0 points [-]

Upvoted for the "Garbage in, Garbage Out" line.

Comment author: lukeprog 12 November 2012 11:52:37AM 4 points [-]

Somehow I managed not to list AI safety promotion in the original draft! Added now.

Comment author: timtyler 17 November 2012 06:13:32PM *  -1 points [-]

Looking many existing risky technologies the consumers and governments are the safety regulators, and manufacturers mostly cater to their demands. Consider the automobile industry, the aeronautical industry and the computer industry for examples.

Comment author: adamisom 17 November 2012 09:40:49PM 0 points [-]

Unfortunately, AGI isn't a "risky technology" where mostly is going to cut it in any sense, including adhering to expectations for safety regulation.

Comment author: timtyler 17 November 2012 10:54:09PM -1 points [-]

All the more reason to use resources effectively. Relatively few safety campaigns have attempted to influence manufacturers. What you tend to see instead are F.U.D. campaigns and neagtive marketing - where organisations attempt to smear their competitors by spreading negative rumours about their products. For example, here is Apple's negative marketing machine at work.

Comment author: Alex_Altair 17 November 2012 09:36:57PM 0 points [-]

Are you suggesting that we encourage consumers to have safety demands? I'm not sure this will work. It's possible that consumers are to reactionary for this to be helpful. Also, I think AI projects will be dangerous before reaching the consumer level. We want AGI researchers to think safe before they even develop theory.

Comment author: timtyler 17 November 2012 10:37:58PM -1 points [-]

It isn't clear that influencing consumer awareness of safety issues would have much effect. However, it suggests that influencing the designers may not be very effective - they are often just giving users the safety level they are prepared to pay for.

Comment author: michaelcurzi 11 November 2012 11:53:37PM 3 points [-]

Yes! It's crucial to have those questions in one place, so that people can locate them and start finding answers.

Comment author: OnTheOtherHandle 14 November 2012 04:08:29AM *  12 points [-]

I’m having trouble deciding where I should target my altruism. I’m basically just starting out in life, and I recognize that this is a really valuable opportunity that I won’t get in the future – I have no responsibilities and no sunk costs yet. For a long time, I’ve been looking into the idea of efficient charity, and I’ve been surfing places like 80K Hours and Give Well. But I get the feeling that I might be too emotionally invested in the idea of donating to, say, disease prevention or inoculation or de-worming (found to be among the most efficient conventional charities) over, say, Friendly AI.

I think that given my skills, personality, etc, there are some good and bad reasons to go into either existential risk mitigation or health interventions, but it’s not like the balance is going to be exactly even. I need some help figuring out what to do – I suppose I could work out some “combination,” but it’s usually not a good idea to hedge your bets in this way because you aren’t making maximum impact.

Direct reasons for health interventions:

  • Lots of good data, an actual dollar amount per life has been calculated; low risk of failure, whereas for everything I’ve read I’m really not sure about where we stand with x-risk or what to do about it or how to calculate if we’ve reduced it

  • Easy to switch from charity to charity and intervention to intervention as new evidence rolls in, whereas x-risk requires some long-term commitments to long-range projects, with only one institution there

  • I would most likely be best off giving money, which I can confidently say I’ll be able to generate; it’s hard to be unclear about whether my actions are making an impact, whereas with x-risk I don't know how much good my actions are creating or whether they're even helping at all

  • Doesn’t require me to have very many more skills than I currently do in order to estimate accurately the costs and benefits

  • It saves lives immediately and continuously, which will be good for motivation and the probability that I will stick it through and actually be altruistic; I also feel a stronger emotional connection to people who are here today rather than potential future humans, although I don’t know if that’s a mistake or a value

Selfish reasons for health interventions:

  • It would make me look virtuous and self-sacrificing when I’m really not hurt very much because there aren’t that many material goods I enjoy

  • Would make me look far less weird and embarrassing at dinner parties

Direct reasons for X-risk:

  • It is by far the highest-stakes problem we have, whereas with health interventions I would be saving lives one by one, in the dozens or the hundreds, if I actually make some sort of marginal impact on x-risk that could translate to many, many more lives

  • Helping to reduce x-risk by helping to bring about Friendly AI would help us increase the probability of all sorts of good things, like the end to all disease and stabilization of the economy, whereas the positive externalities of health intervention are not as dramatic

Selfish reasons for X-risk:

  • I get to feel like I’m on an epic adventure to save the world like all my favorite SF/F heroes

  • It sounds like it would be a pretty awesomely fun challenge and I'd work with cool people

My personal situation: I’m a senior in high school; I’ve read up quite a bit on both developing world health interventions and x-risk. My programming is abysmal, but I can improve quickly because I like it; I’m going into CS hopefully. I’m not the most highly motivated of people – to be quite honest, I’m a procrastinator and would make less of a marginal impact working for SIAI, than some other readers of LW. (That’s where I stand now, but I want to improve my productivity.)

I would do well in a low-stress, medium high pay job through which I could donate money. I harbor the dream of having a really “cool” job, but it’s not necessarily a priority – I can trade awesomeness and challenge for security and ease, as long as my donated money goes to the most useful cause.

I don’t know what I should do. Low risk, low impact/High risk, high impact/Some specific combination? Any variables I'm missing here? I'd love some data that would make the choice more clear.

Comment author: lukeprog 15 November 2012 01:17:18AM 7 points [-]

Sounds like you should ask for a call with the philanthropic career experts at 80,000 Hours if you haven't already.

Comment author: Mitchell_Porter 16 November 2012 04:38:21PM 3 points [-]

I suggest that you practice thinking of yourself as a future member of the posthuman ruling class. In this century, the planet faces the pressures of 9 or 10 billion people trying to raise their standards of living, amid ecological changes like global warming, and at the same time biotechnology, neurotechnology, and artificial intelligence will come tumbling out of Pandora's box. The challenges of the future are about posthuman people living in a postnatural world, and many of the categories which inform current thinking about "efficient charity" and "existential risk" are liable to become obsolete.

Comment author: RyanCarey 12 November 2012 07:26:34AM 10 points [-]

I was reading through these publications one by one, thinking that there must be a quick way to download all pdf links from a page at once, and it turns out there is

Comment author: incariol 12 November 2012 07:42:41PM 9 points [-]

When someone proposes what we should do, where by we he implicitly refers to a large group of people he has no real influence over (as in the banning AGI & hardware development proposal), I'm wondering what is the value of this kind of speculation - other than amusing oneself with a picture of "what would this button do" on a simulation of Earth under one's hands.

As I see it, there's no point in thinking about these kind of "large scale" interventions that are closely interweaved with politics. Better to focus on what relatively small groups of people can do (this includes, e.g. influencing a few other AGI development teams to work on FAI), and in this context, I think out best hope is in deeply understanding the mechanics of intelligence and thus having at least a chance at creating FAI before some team that doesn't care the least about safety dooms us all - and there will be such teams, regardless of what we do today, just take a look at some of the "risks from AI" interviews...

Comment author: ChristianKl 16 November 2012 05:02:08PM *  2 points [-]

When someone proposes what we should do, where by we he implicitly refers to a large group of people he has no real influence over

I'm not sure whether that's true. Government officials who are tasked with researching future trend might right the article. Just because you yourself have no influence on politics doesn't mean that the same is true for everyone who reads the article.

Even if you think that at the moment nobody with policial power reads LessWrong, it's valuable to signal status. If you want to convince a billionaire to fund your project it might be be benefitial to speak about options that require a high amount of resources to pull off.

Comment author: Bruno_Coelho 13 November 2012 01:17:50AM 0 points [-]

In early stages is not easy to focus directly in the organizations x or y, mostly because a good amount of researchers are working in projects who could end in a AGI expert in numerous specific domains. Futhermore, large scale coordination is important too, even if not a top priority. Slowing down a project or funding another is a guided intervetion who could gain some time while technical problems remain unsolved.

Comment author: Pablo_Stafforini 12 November 2012 02:25:59PM *  5 points [-]

Suppose you think that reducing the risk of human extinction is the highest-value thing you can do. Or maybe you want to reduce "x-risk" because you're already a comfortable First-Worlder like me and so you might as well do something epic and cool, or because you like the community of people who are doing it already, or whatever.

I think this post is great: important, informative, concise, and well-referenced. However, my impression is that the opening paragraph trivializes the topic. If you were listing the things we could do to reduce or eliminate global poverty, would you preface your article by saying that "reducing global poverty is cool"? You probably wouldn't. Then why write that kind of preface when the subject is existential risk reduction, which is even more important?

Comment author: Dorikka 13 November 2012 06:51:55PM 6 points [-]

Hm. It's possible that I don't have an good model of people with things like this, but it seems likely that at least some of the people contributing to x-risk reduction might do it for one of these reasons, and this paragraph is making it abundantly clear that the author isn't going to be a jerk about people not supporting his cause for the right reasons. I liked it.

Comment author: ciphergoth 15 November 2012 02:59:07PM 4 points [-]

I took that as anticipating a counter of "Hah, you think your donors really believe in your cause, when really loads of them are just trying to be cool!" - "That's fine, I've noticed their money works just as well."

Comment author: [deleted] 13 November 2012 12:50:53PM 0 points [-]

I took that to be slightly tongue-in-cheek.

Comment author: John_Maxwell_IV 12 November 2012 04:55:18AM *  5 points [-]

Note that if you mostly contribute to meta work, you want to also donate a small sum (say, $15/mo) to strategy work or direct work.

I suspect the mechanism underlying this is a near/far glitch that's also responsible for procrastination. I've had the experience of putting important but deadline-free stuff off for months, sincerely believing that I would get to it at some point (no, I wasn't short on free time...)

It takes time to develop the skill of noticing that now is a good time to do that thing you've been telling yourself to do, and then actually biting the bullet and doing it.

Comment author: Benja 11 November 2012 11:48:37PM *  5 points [-]

I don't currently know of a group pushing (...) for banning AGI development. You could accelerate AGI by investing in AGI-related companies (...)

This is not meant as a criticism of the post, but it seems like we should be able to do better than having some of us give money to groups pushing for banning AGI development, and others invest in AGI-related companies to accelerate AGI, especially if both of these are altruists with a reasonably similar prior aiming to reduce existential risk...

(Both giving to strategic research instead seems like a reasonable alternative.)

Comment author: lukeprog 12 November 2012 01:18:51AM 8 points [-]

Right... it's a bit like in 2004 when my friend insisted that we both waste many hours to go vote on the presidential election, even though we both knew we were voting for opposite candidates. It would have been wiser for us both to stay home and donate to something we both supported (e.g. campaign finance reform), in whatever amount reflected the value of the time we actually spent voting.

I should note that investing in an AGI company while also investing in AGI safety research need not be as contradictory as it sounds, if you can use your investment in the AGI company to bias its development work toward safety, as Legg once suggested. In fact, I know at least three individuals (that I shall not name) who appear to be doing exactly this.

Comment author: TrickBlack 20 November 2012 09:00:28AM 4 points [-]

So I read the title and thought you mean the risk of AI having existential crises... which is an interesting question, when you think about it.

Comment author: John_Maxwell_IV 12 November 2012 04:56:59AM 4 points [-]

Your "caged AGIs" link goes to the SI volunteer site.

Comment author: Curiouskid 18 November 2012 04:13:05AM 1 point [-]

I wish I had an AGI volunteer.

Comment author: lukeprog 12 November 2012 05:28:07AM 1 point [-]

Fixed.

Comment author: heath_rezabek 12 November 2012 06:08:10PM *  5 points [-]

Greetings. New to LessWrong, but particularly compelled by the discussion of existential risk.

It seems like one of the priorities would be to ease the path for people, once they're aware of existential risk, to move swiftly through doing meta work to doing strategy work and direct work. For myself, once I'd become aware of existential risk as a whole, it became an attractor for a whole range of prior ideas and I had to find a way towards direct work as soon as possible. That's easier said than done.

Yet it seems like the shortest path would be to catalyse prosperous industry around addressing the topic. With Bostrom's newer classification scheme, and the inclusion of outcomes such as Permanent Stagnation and Flawed Realization, the problem space is opened wider than if we were forced to deal with a simple laundry list of extinction events.

So: What of accelerating startups, hiring, career paths, and industry around minimizing Permanent Stagnation and Flawed Realization as existential risk subtypes, always with existential risk in mind? I've started an IdeaScale (additions welcomed) along these lines. ie, what activities could accelerate the growth of options for those seeking ways to pour their energy into a livelihood spent mitigating existential risk?

http://vesselcodex.ideascale.com

(The title is after my own work regarding the topic, which has to do with long-term archival and preservation of human potential. I presented this proposal for what I call Vessel Archives at the 100 Year Starship Symposium in September 2012. http://goo.gl/X4Fr9 - Though this is quite secondary to the pressing question of accelerating and incubating ER-reducing livelihood as above.)

Comment author: beoShaffer 13 November 2012 12:40:52AM 5 points [-]

Hi heath_rezabek, welcome to less wrong.

Comment author: wedrifid 12 November 2012 12:21:43AM 3 points [-]

How can I reduce existential risk from AI?

First answer that sprung to my mind: You could work to increase existential risk from other sources. If you make it less likely that an AI will ever be built you reduce the risk of AI. Start work on self-replicating nano-tech or biological weapons. Or even just blow an important building and make it look like an Arabic speaker did it.

That leads to the second solution: When working on AI (or with genies or problem solvers in general) take care construct questions that are not lost purposes.

Comment author: [deleted] 12 November 2012 11:29:37AM 3 points [-]

First answer that sprung to my mind: You could work to increase existential risk from other sources. If you make it less likely that an AI will ever be built you reduce the risk of AI. Start work on self-replicating nano-tech or biological weapons. Or even just blow an important building and make it look like an Arabic speaker did it.

The first two paragraphs in the article prevented that answer from springing to my mind.

Comment author: janusdaniels 18 November 2012 09:29:01PM *  2 points [-]

The investment control software of large financial companies seems the most likely source of rogue AI.

Charlie Stross, in his 2005 novel Accelerando, implicitly suggested "financial device" AI as the most likely seed for rogue AI. Last year, David Brin independently and explicitly promoted the possibility. The idea seems more likely today.

With a multi-million dollar salary from an evil bank, you can contribute to species survival.
You're welcome. And I'm dead serious.

Apparently, Brin first wrote a full outline of the idea here: http://ieet.org/index.php/IEET/more/brin20111217 More recently: http://www.newscientist.com/article/mg21528735.700-artificial-intelligence-to-sniff-out-bankers-scams.html The New Scientist link is from Brin's (excellent) blog, http://davidbrin.blogspot.com/2012/07/bulletins-from-transparency-front.html?showComment=1342734425933#c5447006304917530539

Comment author: [deleted] 12 November 2012 11:27:26AM 2 points [-]

Upvoted for the large number of interesting links.

Comment author: ialdabaoth 11 November 2012 11:41:20PM 2 points [-]

Regarding "making money" / "accumulating wealth": Why is wealth in my hands preferable to wealth in someone else's hands?

Comment author: Pablo_Stafforini 11 November 2012 11:54:48PM 14 points [-]

Because it's extremely unlikely that a random person will be at least as concerned with existential risk as you are.

Comment author: ialdabaoth 12 November 2012 01:07:21AM 5 points [-]

But why is it likely that I'll be better at doing anything about it? Just because I try to be rational, doesn't mean I'm any good at it - especially at something where we have no idea what the correct actions are. How do I know that my efforts will even have a positive dot-product with the "do the right thing" vector?

Comment author: Kaj_Sotala 12 November 2012 10:31:05AM 7 points [-]

The average person has zero interest in fighting existential risk. It's very easy to do better than average, if the average is zero. Even if you've only spent fifty hours (say) familiarizing yourself with the topic, that's already much better than most.

Comment author: [deleted] 12 November 2012 11:32:02AM *  1 point [-]

The average person has zero interest in fighting existential risk.

This strikes me as, ahem, an inflationary use of the term zero. Try negligible instead. :-)

EDIT: Turns out it was an inflationary use of the term average instead. :-) Retracted.

Comment author: Kaj_Sotala 12 November 2012 12:16:27PM 1 point [-]

Well, if we measure interest by what they actually do, then I hold by the "zero".

Comment author: [deleted] 12 November 2012 12:57:30PM *  5 points [-]

EY's interest in fighting existential risks is strictly greater than 0 as far as I can tell; is someone else cancelling that out in the average? (Or by average did you actually mean 'median'?) The number of arms the average human has is strictly less than 2.

Comment author: Kaj_Sotala 12 November 2012 02:18:26PM *  5 points [-]

I meant median.

Comment author: Giles 12 November 2012 01:57:10AM 0 points [-]

Do you feel the odds improve if you choose "become stronger" first?

Comment author: ialdabaoth 12 November 2012 02:31:51AM 2 points [-]

No, because I am perpetually trying to "become stronger", and I still have yet to solve the problem that I'm just "trying" (due to an infinite regress of "trying to try to try to try...)

Comment author: Giles 12 November 2012 02:52:38AM 1 point [-]

What kinds of approach have you tried, and which have resulted in any progress? (at least according to their own metric - e.g. you make progress in "becoming better at math" if you become better at math, even if you're not sure this has improved your general rationality)

Comment author: ialdabaoth 12 November 2012 04:26:44AM 1 point [-]

Well, I've managed to dissuade myself of most cognitive biases clustered around the self-serving bias, by deliberately training myself to second-guess anything that would make me feel comfortable or feel good about myself. This one worked pretty well; I tend to be much less susceptible to 'delusional optimism' than most people, and I'm generally the voice of rational dissent in any group I belong to.

I've tried to make myself more 'programmable' - developed techniques for quickly adjusting my "gut feelings" / subconscious heuristics towards whatever ends I anticipated would be useful. This worked out really well in my 20's, but after awhile I wound up introducing a lot of bugs into it. As it turns out, when you start using meditation and occult techniques to hack your brain, it's really easy to lock yourself into a stable attractor where you no longer have the capacity to perform further useful programming. Oops.

Comment author: Giles 12 November 2012 05:24:29AM 1 point [-]

I'm not sure if this advice will be useful to you, but I think what I'd do in this situation would be to stick to standard techniques and avoid the brain hacking, at least for now. "Standard techniques" might be things like learning particular skills, or asking your friends to be on the lookout for weird behavior from you.

One other thing - though I may have understood you here - you say you've trained yourself not to feel good about yourself in certain circumstances. To compensate, have you trained yourself to feel better about yourself in other circumstances? I'd guess there's an optimal overall level of feeling good about yourself and our natural overall level is probably about right (but not necessarily right about the specifics of what to feel good about)

Comment author: ialdabaoth 12 November 2012 05:48:05AM *  3 points [-]

No, in general I've trained myself to operate, as much as possible, with an incredibly lean dopamine mixture. To hell with feeling good; I want to be able to push on no matter how bad I feel.

(As it turns out, I have limits, but I've mostly trained to push through those limits through shame and willpower rather than through reward mechanisms, to the point that reward mechanisms generally don't even really work on me anymore - at least, not to the level other people expect them to).

A lot of this was a direct decision, at a very young age, to never exploit primate dominance rituals or competitive zero-sum exchanges to get ahead. It's been horrific, but... the best metaphor I can give is from a story called "Those Who Walk Away From Omelas".

Essentially, you have a utopia, that is powered by the horrific torture and suffering of a single innocent child. At a certain age, everyone in the culture is explained how the utopia works, and given two choices: commit fully to making the utopia worth the cost of that kid's suffering, or walk away from utopia and brave the harsh world outside.

I tried to take a third path, and say "fuck it. Let the kid go and strap me in."

So in a sense, I suppose I tried to replace normal feel-good routines for a sort of smug moral superiority, but then I trained myself to see my own behavior as smug moral superiority so I wouldn't feel good about it. So, yeah.

Comment author: Giles 12 November 2012 06:13:38AM 4 points [-]

Are you sure this is optimal? You seem to have goals but have thrown away three potentially useful tools: reward mechanisms, primate dominance rituals and zero-sum competitions. Obviously you've gained grit.

Comment author: Strange7 13 November 2012 10:10:36PM 1 point [-]

Problem being that Omelas doesn't just require that /somebody/ be suffering; if it did, they'd probably take turns or something. It's some quality of that one kid.

Comment author: roystgnr 13 November 2012 11:16:23PM 1 point [-]

Even if benthamite's excellent answer wasn't true, and every random person was at least as concerned with existential risk as you are, it would still be useful for you to accumulate wealth. The economy is not a zero-sum game; producing wealth for yourself does not necessitate reducing the wealth in the hands of others.

Comment author: Benja 11 November 2012 11:50:10PM 1 point [-]

It isn't... if those other hands would give at least as much of it to x-risk reduction as you will.

Comment author: negamuhia 13 November 2012 08:59:54AM *  1 point [-]

I realize (and I'm probably not alone in this) that I've been implicitly using this {meta-work, strategy-work, direct-work} process to try and figure out where/how to contribute. Thanks for this guide/analysis.

Comment author: gjm 12 November 2012 10:37:58PM 1 point [-]

I conjecture that one of the "community of people" links was meant to go somewhere other than where it currently does. (SIAI?)

Comment author: lukeprog 14 November 2012 01:37:27AM 0 points [-]

Fixed.

Comment author: John_Maxwell_IV 12 November 2012 05:00:12AM *  1 point [-]
Comment author: lukeprog 12 November 2012 05:27:13AM 1 point [-]

Yeah, I don't think much of Lumosity either. Can somebody suggest an alternative link?

Comment author: John_Maxwell_IV 12 November 2012 08:06:19AM 1 point [-]

Maybe this? I haven't looked in to it any.

Comment author: 3p1cd3m0n 01 December 2014 12:14:01AM 0 points [-]

How important is trying to personally live longer for decreasing existential risk? IMO, It seems that most risk of existential catastrophes occurs sooner rather than later, so I doubt living much longer is extremely important. For example, Wikipedia says that a study at the Singularity Summit found that the median date for the singularity occurring is 2040, and one personal gave 80% confidence intervals from 5 - 100 years. Nanotechnology seems to be predicted to come sooner rather than later as well. What does everyone else think?

Comment author: hawkice 01 December 2014 01:56:33AM 0 points [-]

I'm having trouble imagining how risk would ever go down, sans entering a machine-run totalitarian state, so I clearly don't have the same assessment of bad things happening "sooner rather than later". I can't imagine a single dangerous activity that is harder or less dangerous now than it was in the past, and I suspect this will continue. The only things that will happen sooner than later are establishing stable and safe equilibria (like post-Cold War nuclear politics). If me personally being alive meaningfully effects an equilibrium (implicit or explicit) then Humanity is quite completely screwed.

Comment author: 3p1cd3m0n 04 December 2014 01:45:22AM 0 points [-]

For one, Yudkowsky in Artificial Intelligence as a Positive and Negative Factor in Global Risk says that artificial general intelligence could potentially use its super-intelligence to decrease existential risk in ways we haven't thought of. Additionally, I suspect (though I am rather uninformed on the topic) that Earth-originating life will be much less vulnerable one it spreads away from Earth, as I think many catastrophes would be local to a single planet. I suspect catastrophes from nanotechnology one such catastrophe.

Comment author: 3p1cd3m0n 20 November 2014 03:31:22AM *  0 points [-]

Are there any decent arguments saying that working on trying to develop safe AGI would increase existential risk? I've found none, but I'd like to know because I'm considering developing AGI as a career.

Edit: What about AI that's not AGI?

Comment author: ike 20 November 2014 02:39:48PM 1 point [-]
Comment author: 3p1cd3m0n 21 November 2014 02:30:03AM 0 points [-]

Thanks. That really helps. Do you know of any decent arguments suggesting that working on trying to develop safe tool AI (or some other non-AGI AI) would increase existential risk?

Comment author: [deleted] 12 November 2012 01:10:25AM 0 points [-]

If you only contribute to meta work for a while, the outside view (around SI, anyway) suggests there's a good chance you'll forget to ever do anything non-meta.

Keeping a to-do list may be a cheaper way of keeping yourself from forgetting.

Comment author: lukeprog 12 November 2012 02:10:58AM 8 points [-]

I don't think it's actually a problem of "forgetting"; I should probably clarify that language. It's more about habit formation. If one takes up the habit of doing no direct work day after day, it may be difficult to break that habit later.

Comment author: [deleted] 12 November 2012 03:37:56AM 4 points [-]

Yes, good point.