LESSWRONG
LW

Comment Permalink

Convince programmers to refuse to work on risky AGI projects:

Please provide constructive criticism.

We're in an era where the people required to make AGI happen are in so much demand that if they refused to work on an AGI that wasn't safe, they'd still have plenty of jobs left to choose from. You could convince programmers to adopt a policy of refusing to work on unsafe AGI. These specifics would be required:

Make sure that programmers at all levels have a good way to determine whether the AGI they're working on has proper safety mechanisms in place. Sometimes employees get such a small view of their job and will be told such confident fluff by management, that they have no idea what is going on. I am not qualified to do this, but if someone reading this post is, it might be very important if you write some guidelines for how programmers can tell whether the AGI they're working on might be unsafe from within their employment position. It may be more effective to give them a confidential hotline. Things can get complicated, both in programming, and in corporate culture, and employees may need help sorting out what's going on.
You could create resources to help programmers organize a strike or programmer's walk. Things like: An anonymous web interface where people interested in striking can post their intent - this would help momentum build. A place for people to post stories about how they took action against unsafe AI projects. They might not know how to organize otherwise (especially in large projects) or might need the inspiration to get moving.
If a union is formed around technological safety, the union could make demands that outside agencies must be allowed to check on the project, and that the company must be forthcoming with all safety related information.

On the feasibility of getting through to the programmers

10 Heading off a near-term AGI arms race

by lincolnquirk

22nd Aug 2012

1 min read

10

I know people have talked about this in the past, but now seems like an important time for some practical brainstorming here. Hypothetical: the recent $15mm Series A funding of Vicarious by Good Ventures and Founders Fund sets off a wave of $450mm in funded AGI projects of approximately the same scope, over the next ten years. Let's estimate a third of that goes to paying for man-years of actual, low-level, basic AGI capabilities research. That's about 1500 man-years. Anything which can show something resembling progress can easily secure another few hundred man-years to continue making progress.

Now, if this scenario comes to pass, it seems like one of the worst-case scenarios -- if AGI is possible today, that's a lot of highly incentivized, funded research to make it happen, without strong safety incentives. It seems to depend on VCs realizing the high potential impact of an AGI project, and of the companies having access to good researchers.

The Hacker News thread suggests that some people (VCs included) probably already realize the high potential impact, without much consideration for safety:

...I think this exactly the sort of innovation timeline real venture capitalists should be considering - funding real R&D that could have a revolutionary impact even if the odds are against it.

The company to get all of this right will be the first two trillion dollar company.

Is there any way to reverse this trend in public perception? Is there any way to reduce the number of capable researchers? Are there any other angles of attack for this problem?

I'll admit to being very scared.

Personal Blog

10

New Comment

70 comments, sorted by

top scoring

Click to highlight new comments since: Today at 7:52 PM

[-]Daniel_Burfoot12y140

That's about 1500 man-years.

1.5e3 is not large compared to the total number of man-years spent on AI, which is probably more like 1.5e5. There are probably 1e4 researchers in AI-related fields, so we're producing at least 1e4 man-years of effort per year. It may be that private sector projects are more promising/threatening than academic projects, but it seems implausible that this would be a 100x effect.

2atucker12y

AI-related fields and AGI related fields are very different in terms of P(uFAI). For the most part, narrow AI and machine learning don't overlap that much with AGI theory in the way that say, AIXI does.

5Kaj_Sotala12y

Interesting characterization - my hunch would have been that AIXI is an interesting thought experiment but ultimately of little to no practical value, while machine learning research seems to be uncovering all kinds of domain-general ways of learning and reasoning.

2Nornagest12y

My experience with applied machine learning is strictly undergraduate-level modulo a little tinkering and a little industry experience, so these impressions might be quite unlike those of an actual specialist, but my sense is that while it comes up with a lot of interesting stuff that might potentially be useful in making a hypothetical AGI, it ultimately isn't that interested in generalizing outside domain-specific approaches and that limits its bandwidth to a large extent. Machine-learning algorithms are treated as -- not exactly a black box, but pretty well distinguished from the task-level inputs and outputs. For example, you might have a pretty clever neural-network variation that no one's ever used before, but most of the actual work in the associated project is probably going to go into highly specialized preprocessing to render down inputs into an easily digestible form. And that's going to do you exactly no good at all if you want to use the same technique on a different class of inputs. (This can be a little irritating for non-AI people too, by the way. An old coworker of mine has a lengthy rant about how all the dominant algorithms for a particular application permute the inputs in all kinds of fantastically clever ways but then end with "and then we feed it into a neural network".)

5Kaj_Sotala12y

I would agree that the specific applications that machine learning generally pursues are useless for general AI, but the general theory that they develop and use (e.g. probabilistic networks, support vector machines, various clustering techniques, etc. etc....) seems like something that AGI would eventually be built on. Of course, the narrow applications get more funding than the general theory, but that's how it always is. My knowledge/experience of ML is probably even less than yours, though. I have this (OpenCog-influenced) mental image of a superintelligent AGI equipped with a huge arsenal of various reasoning and analysis techniques, and when it encounters a novel problem which it doesn't know how to solve, it'll just throw everything it has on it (prioritizing techniques that have worked on similar problems before) until it starts making progress. (For an "artistic" depiction of the same, see "AI thought process visualization, part II" here.) The hard part of such an AGI would then be mostly in finding a good data format that could efficiently represent the outputs of all those different thought mechanisms, and to balance the interactions of various modules together. (I have no idea of how realistic this vision is, and even less of an idea about how to make such an AGI Friendly.)

2latanius12y

I think it's largely true: the narrow AI "arsenal" currently being developed often comes up with results that seem to be transferable between fields. For example, there is a recent paper that applies the same novel strategy both for image understanding and natural language sentence parsing, both with success. Although you often need lots of tinkering to get state-of-art results, producing the same quality just using a general method without any parameters seems to make a good paper. And while the problem of how to build an AGI is not directly solved by these, we certainly get closer to it using them. (You still need a module to recognize/imagine/process visual data, unless the solution is something really abstract like AIXI...)

[-]lsparrish12y90

By "heading off" I think we should be clear that we are referring to go stones, not some form of sabotage. How can we ensure there will be better safety incentives over the next few decades? That sort of thing.

[-]David_Gerard12y50

It's possible that, if the feasibility just isn't there yet no matter the funding, it'll turn out like nanotechnology - funding for molecule-sized robots that gets spent on chemistry instead. (I wonder what the "instead" would be in this case.)

[-]Eudoxia12y130

Narrow AI and machine learning?

2David_Gerard12y

Sounds about right. With the occasional driverless car, which is really pretty amazing.

2billswift12y

I think a working AGI is more likely to result from expanding or generalizing from a working driverless car than from an academic program somewhere. A program to improve the "judgement" of a working narrow AI strikes me as a much more plausible route to GAI.

4Kaj_Sotala12y

Our evolutionary history would seem to support this view - to a first approximation, it would seem to me like general intelligence effectively evolved by stacking one narrow-intelligence module on top of another. Spiders are pretty narrow intelligence, rats considerably less so.

2JulianMorrison12y

And legoland is built of stacking bricks. But try deriving legoland by generalizing a 2x2 blue square.

2Douglas_Knight12y

Note that the driverless car itself came from "an academic program somewhere."

1Eliezer Yudkowsky12y

There are proverbs about how trying to generalize your code will never get to AGI. These proverbs are true, and they're still true when generalizing a driverless car. I might worry to some degree about free-form machine learning algorithms at hedge funds, but not about generalizing driverless cars.

1MugaSofer12y

There go my wild theories about Cars backstory.

1bogus12y

Fear not. There is actual research being done on making self-driving cars more anthropomorphic, in order to enable better communication with pedestrians.

-1latanius12y

Current narrow AIs are unlikely to generalize into AGI, but they contain parts that can be used to build one :)

0atucker12y

Narrow-AI driverless cars will probably not decide that they need to take over the world in order to get to their destination in the most efficient way. Even if it would be better, I would be very surprised if they decided to model the world that generally for the purposes of driving. There's only so much modeling of the world/general capability you need in order to solve very domain-specific problems.

0billswift12y

The reason for expanding a narrow AI is the same for a tool agent not staying restricted; the narrow domain they are designed to function in is embedded in the complexity of the real world. Eventually someone is going to realize that the agent/AI can provide better service if they understand more about how their jobs fit into the broader concerns of their passengers/users/customers and decide to do something about it.

0atucker12y

AIXI is able to be widely applicable because it tries to model every possible program that the universe could be running, and then it eventually starts finding programs that fit. Driverless cars may start containing modeling things other than driving, and may even start trying to predict where their users are going to be, but I suspect that it would try and just track user habits or their smartphones, rather than trying to figure out their owner's economic and psychological incentives for going to different places. Trying to build a car that's generally capable of driving and figuring out new things about driving might be dangerous, but there's plenty of useful features to give people before they get there. Just wondering, is your intuition coming from the tighter tie to reality that a driverless car would have?

-2Kawoomba12y

"It was terrible, officer ... my mother, she was so happy with her new automatic car! It seemed to anticipate her every need! Even when she forgot where she wanted to go, in her old age, the car would remember and take her there ... she had been so lonely ever since da' passed. I can't even fathom how the car got into her bedroom, or what it was, oh god, what it was ... doing to her! The car, it still ... it didn't know she was already ... all that blood ..."

0jmmcd12y

Has LW, or some other forum, held any useful previous discussion on this topic?

0Manfred12y

Not that I know of, but I'm pretty sure billswift's position does not represent that of most LWers.

3Dolores198412y

It certainly doesn't represent mine. The architectural shortcomings of narrow AI do not lend themselves to gradual improvement. At some point, you're hamstrung by your inability to solve certain crucial mathematical issues.

2billswift12y

You add a parallel module to solve the new issue and a supervisory module to arbitrate between them. There are more elaborate systems that could likely work better for many particular situations, but even this simple system suggests there is little substance to your criticism. See Minsky's Society of Mind, or some papers on modularity in evolutionary psych, for more details.

0Dolores198412y

Sure you can add more modules. Except that then you've got a car-driving module, and a walking module, and a stacking-small-objects module, and a guitar-playing module, and that's all fine until somebody needs to talk to it. Then you've got to write a Turing-complete conversation module, and (as it turns out) having a self-driving car really doesn't make that any easier.

4V_V12y

Do you realize that human intelligence evolved exactly that way? A self-swimming fish brain with lots of modules haphazardly attached.

0Dolores198412y

Evolution and human engineers don't work in the same ways. It also took evolution three million years.

6V_V12y

True enough, but there is no evidence that general intelligence is anything more than a large collection of specialized modules.

0jmmcd12y

I believe you, but intuitively the first objection that comes to my mind is that a car-driving AI doesn't have the same type of "agent-ness" and introspection that an AGI would surely need. I'd love to read more about it.

4roystgnr12y

Best case scenario, it'll turn out like space travel: something that we "did already" but that wasn't nearly as interesting as all those wild-eyed dreamers hoped. I don't see that happening in this context, though; with space travel, we "cheated" our way to a spectacular short-term goal by using politically motivated blank checks while ignoring longer-term economics. Competing venture capitalists are less likely to ignore long-term economics, and any "cheating" is likely to mean shortcuts with regards to safety, not sustainability.

0Luke_A_Somers12y

That's a heck of a condition, and this condition failing seems like our best hope for survival, if the 'spirit' of the original hypothetical holds - that this work ends up really taking off, in practical systems.

[-]palladias12y20

I'm curious at what likelihood of AGI imminence SI or LessWrong readers would think it was a good idea to switch over to an ecoterrorist strategy. The day before the badly vetted machine is turned on is probably a good day to set the charges to blow during the night shift. The funding of this project is probably too early.

Do people think SI should be devoting more of its time and resources to corporate espionage and/or sabotage if unfriendly AI is the most pressing existential threat?

[-][anonymous]12y120

I vote that criminal activity shouldn't be endorsed in general.

8Vaniver12y

On first reading, I read your name as "jailbot," which seemed pretty appropriate for this comment.

7A1987dM12y

Discussions of illicit drugs or ways of getting copyrighted material without the consent of the copyright holder aren't unprecedented on LW.

[-]Risto_Saarelma12y160

With the difference that many people think it may have been a mistake to make those things illegal to begin with. People considering industrial sabotage to stop UFAI probably don't think that industrial sabotage should be legal in general.

4palladias12y

I see this question as analogous to the discussion in the Brain Preservation Foundation thread about whether not donating reveals preferences or exposes belief in belief. Why is asking about non-lethal sabotage too qualitatively different to get at the same question?

4palladias12y

Because it's bad tactics to endorse it in the open or because sabotaging unfriendly AI research is a case of not even if it's the right thing to do? I assume you'd slow down or kibosh a not-proved-to-be-friendly AGI project if you had the authority to do so. But you wouldn't interfere if you didn't have legitimate authority over the project? There are plenty of not-illegal, but still ethical norm-breaking opportunities for sabotage, (deny the people on the project tenure, if you're in a university, hire the best researchers away, etc). Do you think this shouldn't be discussed out of respect for the law, out of respect for the autonomy of researchers, or a mix of both?

8Xachariah12y

If we were certain a uFAI were going online in a matter of days, it would be everyone's responsibility to stop it by any means possible. Imminent threat to humanity and all that. However, it's a very low probability that it'll ever get to that point. Talking about and endorsing (hypothetical) unethical activity will impose social costs in the meanwhile. So, it's a net negative to discuss it.

0Vladimir_Nesov12y

What specifically do you consider low probability? That an uFAI will ever be launched, or that there will be an advance high credibility warning?

[-]gwern12y130

I'd argue the latter. It's hard to imagine how you could know in advance that a uFAI has a high chance of working, rather than being one of thousands of ambitious AGI projects that simply fail.

(Douglas Lenat comes to you, saying that he's finished a powerful fully general self-modifying AI program called Eurisko, which has done very impressive things in its early trials, so he's about to run it on some real-world problems on a supercomputer with Internet access; and by the way, he'll be alone all tomorrow fiddling with it, would you like to come over...)

2Xachariah12y

Sorry, I was imprecise. I consider it likely that eventually we'll be able to make uFAI, but unlikely that any particular project will make uFAI. Moreover, we probably won't get appreciable warning for uFAI because if researchers knew they were making a uFAI then they wouldn't make one. Thus, we have to adopt a general strategy that can't target any specific research group. Sabotage does not scale well, and would only drive research underground while imposing social costs on us meanwhile. The best bet then is to promote awareness of uFAI risks and try to have friendliness theory completed by the time the first AGI goes online. Not surprisingly, this seems to be what SIAI is already doing. Discussion of sabotage just harms that strategy.

6jmmcd12y

It might damage LW's credibility among decision-makers and public opinion. Of course, it might improve LW's credibility among certain other groupings. PR is a tricky balancing act. PETA is a good example in both cases.

0V_V12y

Indeed. Comment like those increase my belief that LW is home to some crazy dangerous doomsday cult that stores weapons caches and prepares terror attacks. (I still don't assign an high probability to that belief, yet, but still higher than most communities I know) I came here from OB, and I lurked a bit before posting precisely because I didn't like these kind of undertones. If that attitude becomes more prevalent I will probably go away to avoid any association.

1enoonsti12y

I was going to say: "Well, on the bright side, at least your username is not really Googleable." Then I Googled it just for fun, and found you on the first page of the results (゜レ゜)

0Bruno_Coelho12y

Intelligence does not imply benevolence. Surely, there already are people who will try to sabotage unFriendly projects.

0IlyaShpitser12y

I don't think you quite understand the hammer that will come down if anything comes of your questions. Nothing of what you built will be left. I don't think many non-illegal sabotage avenues are open to this community. You can't easily influence the tenure process, and hiring the best researchers is notoriously difficult, even for very good universities/labs. ---------------------------------------- Re: OP, I think you are worried over nothing.

0palladias12y

That's why I asked whether Less Wrongers would prefer SI to devote more of it's time to slowing down other people's unfriendly AI relative to how much time it spends constructing FAI. I agree, SI staff shouldn't answer.

[-]IlyaShpitser12y100

I think any sequence of events that leads to anyone at all in any way associated with either lesswrong or SI doing anything to hinder any research would be a catastrophe for this community. At best, you will get a crank label (more than now, that is), at worst the FBI will get involved.

7David_Gerard12y

I think you may be a bit late.

-4Xachariah12y

Yes. It's much better to tile the universe with paperclips than to have this community looked on poorly. How ever could he have gotten his priorities so crossed?

-2Epiphany12y

If there is a big enough AI project out there, especially if it will be released as freeware, others won't work on it. That would be high-risk and result in a low return on investment. Three ideas to prevent unfriendly AGI (Scroll to "Help good guys beat the arms race") Also, I don't think my other two risky AGI deterring ideas aren't do-able simultaneously. Not sure how many people it would take to get those moving on a large enough scale, but it's probably nowhere near as much as making a friendly AGI.

-2Epiphany12y

Three legal ideas to prevent risky AGI projects Sabotage would probably backfire: Why sabotaging unfriendly AGI wouldn't work

9David_Gerard12y

This question has already been raised on LW.

2palladias12y

Merci!

-1Epiphany12y

Sabotage would not work. Several reasons: * If the people working on AGI projects fear sabotage, they'll just start working on them in private. Then, you'd be lulled into a sense of complacency, thinking they're not working on it. You will fail to take action, even legal ones. Then, one day, the AGI will be released, and it will be too late. * Anybody who sets out to make an AGI without first putting a lot of thought into safety is either really risk-taking, really stupid, or really crazy. People who are big risk-takers, really stupid, or really crazy do not respond like normal people do to threats. This would be really, really ineffective on them. You can look up the recidivism rate for people who are put into jail and see that there are a great many people who are not stopped by punishment. If you do some research on the kinds of risks business people take, you'll see the same risk-taking attitude, only in a more constructive form. * People who previously didn't know anything about the AGI project would view the AGI company as the good guys and the saboteurs as the bad guys. Whenever violence gets involved, opinions polarize, permanently. The saboteurs would find it 10,000 times harder just to exist, they'd lose a huge amount of support, making it 10,000 times harder to exist again. MUCH WORSE would be the biasing effects of "stimulus generalization". That is the psychological effect that causes people to feel prejudiced against, say Middle Easterners in general because they fear terrorists. If those who want to protect the world from risky AGI begin to sabotage projects, public opinion may just round off and lump everybody warning against risky AGI under the "terrorist" label. They might feel such a strong bias that they assume anyone warning about unfriendly AGI is a psycho and ignore the danger of unfriendly AGI completely * There will be numerous projects in numerous places at numerous times. It wouldn't be a matter of stopping just one project, or pro

0MugaSofer12y

It's possible you've partially misunderstood the purpose of this idea; such sabotage would not be a deterrent to be publicised, but a tactic to permenantly derail any unFAI that nears completion.

[-]Epiphany12y10

Convince programmers to refuse to work on risky AGI projects:

Please provide constructive criticism.

Make sure that programmers at all levels have a good way to determine whether the AGI they're working on has proper safety mechanisms in place.

... (read more)

-1Epiphany12y

Gwern responded to my comment in his Moore's Law thread. I don't know why he responded over there instead of over here but I decided that it was more organized to relocate the conversation to the comment it is about so I put my response to him here. Do you have evidence one way or the other of what proportion of programmers get the existential risk posed by AGI? In any case, I don't know how to tell whether you're too pessimistic or whether I am too optimistic here. researches for figures for this project There are between 1,200,000 and 1,450,000 programmers depending on whether you want to count web people (who have been lumped together) in the USA according to the 2010 US Bureau of Labor Statistics. That's not the entire world but getting the American programmers on board would be major progress and researching the figures for all 200 countries in the world is outside the scope of this comment, so I will stick to that for right now. LessWrong has over 13,000 users and over 10,000,000 visits. It isn't clear what percentage of the American programmer population has been exposed to AI and existential risk this way (and a bit over half the visits are from Americans) but since LessWrong has lots of programmers and has eight times as many visits as there are programmers in America, it's possible that a majority of American programmers have at least heard of existential risk or SI. This is just the beginning though because LessWrong is growing pretty fast and it could grow even faster if I (or someone) were to get involved in web marketing such as improving the SEO or improving the site's presentation (I may do both of these, though I want to address the risk of endless September first and I'm letting that one cool off for a while, at Luke's advice, so that people don't explode because I made too many meta threads). I don't see any research on what percentage of programmers believe that AI poses significant risks... I doubt there is any right now, but maybe you know

[-]timtyler12y00

The company to get all of this right will be the first two trillion dollar company.

Is there any way to reverse this trend in public perception?

You don't supply a counter-argument. Do you disagree - or are you looking for a way to create a mass delusion?

[-]Epiphany12y-40

Help good guys beat the race:

Please provide constructive criticism.

An open source project might prevent this problem, not because having an open source AGI is safe, but because 1.) open source projects are open, so anybody can influence it, including people who are knowledgeable about risks and 2.) the people involved in open source projects probably tend to have a pretty strong philanthropic streak and they're more likely to listen to the dangers than a risk-taking capitalist. The reason it may stop them is this: If an open source project gets there fi... (read more)

0Risto_Saarelma12y

Analogizing AGI mainly to existing software projects probably isn't a good starting point for an useful contribution. The big problems are mostly tied to the unique features an actual AGI would have, not to making a generic software project with some security implications work out right. For a different analogy, think about a software that fits on a floppy disk that somehow turns any laptop into an explosive device with a nuclear bomb level yield (maybe it turns out you can set up a very specific oscillation pattern in a multicore CPU silicon that will trigger a localized false vacuum collapse). I'm not sure I'd be happy to settle with "code gets stolen anyway, so let's make sure everyone gets access to it". An actual working AGI could be extremely weaponizable both for very cheap and into something much more dangerous than any software engineering analogy gives reason to suppose, and significantly less useful as a defensive than as an offensive measure.

-2Epiphany12y

Okay. I get that AGI would be this powerful. What I don't get is that the code for it would fit onto a floppy disk. When you say I am making a mistake analogizing AGI to existing software projects, what precisely do you mean to say? Is it that it really wouldn't need very many programmers? Is it that problems with sloppy, rushed coding would be irrelevant? I'm not sure exactly how this counters my point. I'm not happy with it. I think it's better than the alternative. See next point. Agreed. That is precisely why everyone should have it. Because it's "the one ring". They say, "absolute power corrupts absolutely" because there are a billion examples of humans abusing power throughout history. You can't trust anybody with that much power. It will ruin the checks and balances between governments and the people they're supposed to serve, it will ruin the checks and balances between branches of governments and it will make hackers, spies and any criminal or criminal organization who are capable of stealing the software (this might be terrorists, the mafia, gangs, corrupt government leaders, cult leaders, etc.) into superpowers. To check and balance the power there needs to be a mutually assured destruction type threat between the following: The people and the governments they serve. Each branch of governments and the other branches of those governments. The pirates, hackers, spies and criminals and the good people in the world. The reason the US government was set up the way it was - with the right to bear arms and with and balances between branches of government - is because power corrupts and mutually assured destruction keeps the humans accountable, and this type of accountability is necessary to keep the system healthy. In a world where AGI exists, the right to bear arms needs to include AGI, or power imbalances will probably ruin everything. We can't assume the AGIs will all be friendly. Even if we succeed in the incredibly hard task of making sure every AGI

[-]Epiphany12y-40

Create public relations nightmare for anyone producing risky AGI:

Please provide constructive criticism.

One powerful way to get people thinking about safety is if clever ways are invented to shout from the rooftops that this could be dangerous and present the message in a way that most people will grok. If everybody is familiar enough with how dangerous it could be, then funding an AGI project without a safety plan in place would be a PR disaster for the companies doing it. That would put a lot of pressure on them to put safeties into place. This wouldn'... (read more)

[-]Epiphany12y-40

Even Faster Solution:

Survey a bunch of open source people asking them if they'd switch to working on friendly AGI in the event that an AGI project started without enough safety, or get their signatures. Surely the thousands of programmers now working on projects like Firefox and Open Office, who clearly have an altruistic bent as they are working for free, will see that it is more important to prevent unfriendly AGI than to make sure the next version of these smaller projects are released on time.

If we can honestly say to these companies "If you tr... (read more)

[+]Epiphany12y-60

Moderation Log

70Comments