= 27d567bc2d48d9f40e9ccc20ea370b15)
Google may be trying to take over the world
So I know we've already seen them buying a bunch of ML and robotics companies, but now they're purchasing Shane Legg's AGI startup. This is after they've acquired Boston Dynamics, several smaller robotics and ML firms, and started their own life-extension firm.
Is it just me, or are they trying to make Accelerando or something closely related actually happen? Given that they're buying up real experts and not just "AI is inevitable" prediction geeks (who shall remain politely unnamed out of respect for their real, original expertise in machine learning), has someone had a polite word with them about not killing all humans by sheer accident?
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Comments (133)
Shane Legg is familiar with AI risks. So is Jaan Tallinn, a top donor of MIRI, who is also associated with DeepMind. I suppose they will talk about their fears with Google.
Actually, there does seem to have been a very quiet press release about this acquisition resulting in a DeepMind ethics board.
So that's a relief.
Is there any more information beyond mentions like this?
Not to mention of course the Google employees that post on LW.
I didn't know there were any. My guess is that you have to be pretty high in the hierarchy to actually steer Google into a direction that would suit MIRI (under the assumption that people who agree with MIRI are in the minority).
Hi!
plus cousin_it and at least 2-3 others. Plus Ctrl+F for Google here http://intelligence.org/team/. Moshe Looks might be one of Google's AGI people I think.
Greetings from Dublin! You're right that the average employee is unlikely to matter, though.
Eliezed specifically mentioned Google in his Intelligence Explosion Microeconomics paper as the only named organization that could potentially start an intelligence explosion.
In these interviews Larry Page gave years ago he constantly said that he wanted Google to become "the ultimate search engine" that would be able to understand all the information in the world. And to do that, Larry Page said, it would need to be 'true' artificial intelligence (he didn't say 'true', but it comes clear what he means in the context).
Here's a quote by Larry Page from the year 2007:
I doubt it would be very Friendly if you use MIRI's definition, but it doesn't seem like they have something 'evil' in their mind. Peter Norvig is the co-author of AI: A Modern Approach, which is currently the dominant textbook in the field. The 3rd edition had several mentions about AGI and Friendly AI. So at least some people in Google have heard about this Friendliness thing and paid attention to it. But the projects run by Google X are quite secretive, so it's hard to know exactly how seriously they take the dangers of AGI and how much effort they put into these matters. It could be, like lukeprog said in October 2012, that Google doesn't even have "an AGI team".
Not that I know of, anyway. Kurzweil's team is probably part of Page's long-term AGI ambitions, but right now they're focusing on NLP (last I heard). And Deep Mind, which also has long-term AGI ambitions, has been working on game AI as an intermediate step. But then again, that kind of work is probably more relevant progress toward AGI than, say, OpenCog.
IIRC the Deep Mind folks were considering setting up an ethics board before Google acquired them, so the Google ethics board may be a carryover from that. FHI spoke to Deep Mind about safety standards a while back, so they're not totally closed to taking Friendliness seriously. I haven't spoken to the ethics board, so I don't know how serious they are.
Update: "DeepMind reportedly insisted on the board’s establishment before reaching a deal."
Update: DeepMind will work under Jeff Dean at Google's search team.
And, predictably:
...despite the fact that AI systems already fly planes, drive trains, and pilot Hellfire-carrying aerial drones.
NYTimes also links to LessWrong.
Quote:
It would be quite a reach to insist that we need to worry about the ethics of the control boards which calculate how to move elevons or how much to open a throttle in order to maintain certain course or speed. Autonomous UAVs able to open fire without a human in the loop are much more worrying.
I imagine that some of the issues the ethics board might have to deal with eventually would be related to self-agentizing tools, in Karfnofsky-style terminology. For example, if a future search engine receives queries whose answers depend on other simultaneous queries, it may have to solve game-theoretical problems, like optimizing traffic flows. These may some day include life-critical decisions, like whether to direct drivers to a more congested route in order to let emergency vehicles pass unimpeded.
They actually link to LessWrong in the article, namely to my post here.
I personally suspect the ethics board exists for more prosaic reasons. Think "don't bias the results of people's medical advice searches to favor the products of pharmaceutical companies that pay you money" rather than "don't eat the world".
EDIT: just saw other posts including quotes from the head people of the place that got bought. I still think that this is the sort of actual issues they will deal with, as opposed to the theoretical justifications.
So, to summarize, Google wants to build a potentially dangerous AI, but they believe they can keep it as an Oracle AI which will answer questions but not act independently. They also apparently believe (not without some grounding) that true AI is so computationally expensive in terms of both speed and training data that we will probably maintain an advantage of sheer physical violence over a potentially threatening unboxed oracle for a long time.
Except that they are also blatant ideological Singulatarians, so they're working to close that gap.
Comment by Juergen Schmidhuber:
Yes, or in other words, these are the competent AGI researchers.
Buying off AGI startups and then letting the relevant programmers program smart cars seems to me a quite good move to stall UFAI.
Can you elaborate?
It sets AGI-minded programmers (under circumstances expected to yield UFAI) onto tasks that would not be expected to result in AGI of any sort (driving)
I get that part. Is there some reason I'm missing as to why Google wouldn't utilize the talent at DeepMind to pursue AGI-relevant projects?
I mean, Google has great resources (much more than MIRI or anyone else) and a proven record of success at being instrumentally rational in the techincal/programming arena (i.e. winning on a grand scale for a length of time). They are adding folks who, from what I read on LW, actually understand AGI's complexity, implications, etc.
Just nervousness about UFAI.
This analysis seems to be based on AGI-mindedness being an inherent property of programmers, and not a response to market forces.
No... not at all! Quite the opposite, in fact. If it were inherent, then moving them away from it would be ineffective.
Microsoft seems to focus on AI as well:
Q: You are in charge of more than 1000 research labs around the world.
What kind of thing are you focusing on?
Microsoft: A big focus right now, really on point for this segment, is artificial intelligence.
We have been very focused.
It is our largest investment area right now.
Peter Norvig is at least in principle aware of some of the issues; see e.g. this article about the current edition of Norvig&Russell's AIAMA (which mentions a few distinct way in which AI could have very bad consequences and cites Yudkowsky and Omohundro).
I don't know what Google's attitude is to these things, but if it's bad then either they aren't listening to Peter Norvig or they have what they think are strong counterarguments, and in either case an outsider having a polite word is unlikely to make a big difference.
Peter Norving was a resident at Hacker School while I was there, and we had a brief discussion about existential risks from AI. He basically told me that he predicts AI won't surpass humans in intelligence by so much that we won't be able to coerce it into not ruining everything. It was pretty surprising, if that is what he actually believes.
My guess is that most people at Google, who are working on AI, take those risks somewhat seriously (i.e. less seriously than MIRI, but still acknowledge them) but think that the best way to mitigate risks associated with AGI is to research AGI itself, because the problems are intertwined.
Why do you think you have a better idea of the risks and solutions involved than they do, anyway? Superior AI expertise? Some superior expert-choosing talent of yours?
My suggestion to Google is to free up their brightest minds and tell them to talk to MIRI for 2 weeks, full-time. After the two weeks are over, let each of them write a report on whether Google should e.g. give them more time to talk to MIRI, accept MIRI's position and possibly hire them, or ignore them. MIRI should be able to comment on a draft of each of the reports.
I think this could finally settle the issue, if not for MIRI itself then at least for outsiders like me.
Well, that's sort of like having the brightest minds at CERN spend two weeks full time talking to some random "autodidact" who's claiming that LHC is going to create a blackhole that will devour the Earth. Society can't work this way.
Does that mean there is a terrible ignored risk? No, when there is a real risk, the brightest people of extreme and diverse intellectual accomplishment are the ones most likely to be concerned about it (and various "autodidacts" are most likely to fail to notice the risk).
This is an unusual situation though. We have a lot of smart people who believe MIRI (they are not idiots, you've to grant them that). And you and me are not going to change their mind, ever, and they are hardly going to convince us. But if a bunch of independent top-notch people were to accept MIRI's position, then that would certainly make me assign a high probability to the possibility that I simply don't get it and that they are right after all.
In the case of the LHC, independent safety reviews have been conducted. I wish this was the case for the kinds of AI risk scenarios imagined by MIRI.
If you pitch something stupid to a large enough number of smart people, some small fraction will believe.
Not for every crackpot claim. edit: and since they got an ethical review board, that's your equivalent of what was conducted...
There's a threshold. Some successful trading software, or a popular programming language, or some AI project that does something world-level notable (plays some game really well for example), that puts one above the threshold. Convincing some small fraction of smart people does not. Shane Legg's startup evidently is above the threshold.
As for the risks, why would you think that Google's research is a greater risk to mankind than, say, MIRI's? (assuming that the latter is not irrelevant, for the sake of the argument)
If MIRI was right then, as far as I understand it, a not quite friendly AI (broken friendly AI) could lead to a worse outcome than a general AI that was designed without humans in mind. Since in the former case you would end up with something that keeps humans alive, but e.g. gets a detail liked boredom wrong, while in the latter case you would be transformed into e.g. paperclips. So from this perspective, if MIRI was right, it could be the greater risk.
Well, the other issue is also that people's opinions tend to be more informative of their own general plans than about the field in general.
Imagine that there's a bunch of nuclear power plant engineering teams - before nuclear power plants - working on different approaches.
One of the teams - not a particularly impressive one either - claimed that any nuclear plant is going to blow up like a hundred kiloton nuclear bomb, unless fitted with a very reliable and fast acting control system. This is actually how nuclear power plants were portrayed in early science fiction ("Blowups Happen", by Heinlein).
So you look at the blueprints, and you see that everyone's reactor is designed for a negative temperature coefficient of reactivity, in the high temperature range, and can't blow up like a nuke. Except for one team whose reactor is not designed to make use of a negative temperature coefficient of reactivity. The mysterious disagreement is explained, albeit in a very boring way.
Except that this contrarian team, made of high school drop-outs, former theologians, philosophers, mathematicians and coal power station technicians, never produce an actual design, instead they spend all their time investigating arcane theoretical questions about renormalization in quantum field theory and publish their possibly interesting results outside the scientific peer review system, relying on hype to disseminate them.
Well, they still have some plan, however fuzzy it is. The plan involves a reactor which according to it's proponents would just blow up like a 100 kiloton nuke if not for some awesome control system they plan to someday work on. Or in case of AI, a general architecture that is going to self improve and literally kill everyone unless a correct goal is set for it. (Or even torture everyone if there's a minus sign in the wrong place - the reactor analogy would be a much worse explosion still if the control rods get wired backwards. Which happens).
My feeling is that there may be risks for some potential designs, but they are not like "the brightest minds that build the first AI failed to understands some argument that even former theologians can follow" (In fiction this happens because said theologian is very special, in reality it happens because the argument is flawed or irrelevant)
This is related to something that I am quite confused about. There are basically 3 possibilities:
(1) You have to be really lucky to stumble across MIRI's argument. Just being really smart is insufficient. So we should not expect whoever ends up creating the first AGI to think about it.
(2) You have to be exceptionally intelligent to come up with MIRI's argument. And you have to be nowhere as intelligent in order to build an AGI that can take over the world.
(3) MIRI's argument is very complex. Only someone who deliberately thinks about risks associated with AGI could come up with all the necessary details of the argument. The first people to build an AGI won't arrive at the correct insights in time.
Maybe there is another possibility on how MIRI could end up being right that I have not thought about, let me know.
It seems to me that what all of these possibilities have in common is that they are improbable. Either you have to be (1) lucky or (2) exceptionally bright or (3) be right about a highly conjunctive hypothesis.
Can you cite some evidence for this?
Um, surely if you take (a) people with a track record of successful achievement in an area (b) people without a track record of success but who think they know a lot about the area, the presumption that (a) is more likely to know what they're talking about should be the default presumption. It may of course not work out that way, but that would surely be the way to bet.
Yes, I agree, but that is only part of the story, right?
What if autodidacts, in their untutored excitability, are excessively concerned about a real risk? Or if a real risk has nearly all autodidacts significantly worried, but only 20% of actual experts significantly worried? Wouldn't that falsify /u/private_messaging's assertion? And what's so implausible about that scenario? Shouldn't we expect autodidacts' concerns to be out of step with real risks?
If autodidacts are excessively concerned, then why would it be worth for experts to listen to them?
It may not be. I was not taking issue with the claim "Experts need not listen to autodidacts." I was taking issue with the claim "Given a real risk, experts are more likely to be concerned than autodidacts are."
I would assume that experts are likely to be concerned to an extent more appropriate to the severity of the risk than autodidacts are.
There can be exceptions, of course, but when non-experts make widely more extreme claims than experts do on some issue, especially a strongly emotively charged issue (e.g. the End of the World), unless they can present really compelling evidence and arguments, Dunning–Kruger effect seems to be the most likely explanation.
That is exactly what I would assume too. Autodidacts' risk estimates should be worse than experts'. It does not follow that autodidacts' risk estimates should be milder than experts', though. The latter claim is what I meant to contest.
"Autodidacts" was in quotes for a reason.
Let's talk about some woo that you're not interested in. E.g. health risks of thymerosal and vaccines in general. Who's more likely to notice it, some self proclaimed "autodidacts", or normal biochemistry experts? Who noticed the possibility of a nuke, back-then conspiracy theorists or scientists? Was Semmelweis some weird outsider, or was he a regular medical doctor with medical training? And so on and so forth.
Right now, experts are concerned with things like nuclear war, run-away methane releases, epidemics, and so on, while various self proclaimed existential risk people (mostly philosophers) seem to be to greater or lesser extent neglecting said risks in favor of movie plot dangers such as runaway self improving AI or perhaps totalitarian world government. (Of course if you listen to said x-risk folks, they're going to tell you that it's because the real experts are wrong.)
All are good and relevant examples, and they all support the claim in question. Thanks!
But your second paragraph supports the opposite claim. (Again, the claim in question is: Experts are more likely to be concerned over risks than autodidacts are.) In the second paragraph, you give a couple "movie plot" risks, and note that autodidacts are more concerned about them than experts are. Those would therefore be cases of autodidacts being more concerned about risks than experts, right?
If the claim were "Experts have more realistic risk estimates than autodidacts do," then I would readily agree. But you seem to have claimed that autodidacts' risk estimates aren't just wrong--they are biased downward. Is that indeed what you meant to claim, or have I misunderstood you?
To clarify, I have nothing anything against self educated persons. Some do great things. The "autodidacts" was specifically in quotes.
What is implausible, is this whole narrative where you have a risk obvious enough that people without any relevant training can see it (by the way of that paperclipping argument), yet the relevant experts are ignoring it. Especially when the idea of an intelligence turning against it's creator is incredibly common in fiction, to the point that nobody has to form that idea on their own.
In general, current AGI architectures work via reinforcement learning: reward and punishment. Relevant experts are worried about what will happen when an AGI with the value-architecture of a pet dog finds that it can steal all the biscuits from the kitchen counter without having to do any tricks.
They are less worried about their current creations FOOMing into god-level superintelligences, because current AI architectures are not FOOMable, and it seems quite unlikely that you can create a self-improving ultraintelligence by accident. Except when that's exactly what they plan for them to do (ie: Shane Legg).
Juergen Schmidhuber gave an interview on this very website where he basically said that he expects his Goedel Machines to undergo a hard takeoff at some point, with right and wrong being decided retrospectively by the victors of the resulting Artilect War. He may have been trolling, but it's a bit hard to tell.
I'd need to have links and to read it by myself.
With regards to reinforcement learning, one thing to note is that the learning process is in general not the same thing as the intelligence that is being built by the learning process. E.g. if you were to evolve some ecosystem of programs by using "rewards" and "punishments", the resulting code ends up with distinct goals (just as humans are capable of inventing and using birth control). Not understanding this, local genuises of the AI risk been going on about "omg he's so stupid it's going to convert the solar system to smiley faces" with regards to at least one actual AI researcher.
Here is his interview. It's very, very hard to tell if he's got his tongue firmly in cheek (he refers to minds of human-level intelligence and our problems as being "small"), or if he's enjoying an opportunity to troll the hell out of some organization with a low opinion of his work.
With respect to genetic algorithms, you are correct. With respect to something like neural networks (real world stuff) or AIXI (pure theory), you are incorrect. This is actually why machine-learning experts differentiate between evolutionary algorithms ("use an evolutionary process to create an agent that scores well on X") versus direct learning approaches ("the agent learns to score well on X").
What, really? I mean, while I do get worried about things like Google trying to take over the world, that's because they're ideological Singulatarians. They know the danger line is there, and intend to step over it. I do not believe that most competent Really Broad Machine Learning (let's use that nickname for AGI) researchers are deliberately, suicidally evil, but then again, I don't believe you can accidentally make a dangerous-level AGI (ie: a program that acts as a VNM-rational agent in pursuit of an inhumane goal).
Accidental and evolved programs are usually just plain not rational agents, and therefore pose rather more limited dangers (crashing your car, as opposed to killing everyone everywhere).
Upvoted for writing style.
I'm quite happy to hear that, but it's not very useful advice. I'm not an AIXI agent, so I can't deduce what's being praised solely from the fact that it is praised.
You can, however, glean information about how to write, particularly given that the reasoning was made explicit. That probably has more actual practical value for just about all readers.
If you believe this, Deepmind had to push for an ethics board which suggests that people are mentioning it to Google and that Google is not taking the issue too seriously.
That interpretation seems tenuous. The sentence in which "pushed" is used:
suggests nothing more than that the proposal originated with DeepMind. I may as well imagine that Google's apparent amenability to the arrangement augurs well.
(Unless the article goes on to explain further? Not a subscriber.)
I'd expect their ethics board will in any case not be about the risk of killing all humans, but about things like privacy issues and the more mundane safety issues that you get when you connect a machine learning thing to a robot (or a car).
I'm sure Google will be right on that.
Well, someone's gotta do it.
A Weyland-Yutani style outcome is a far bigger risk. EDIT: Does this mean anti-trust laws probably should've hit them a long time ago?
Should've, sure. Didn't. And won't, in all likelihood. Google is very, very rich, influential, and popular with the public, so the chances of them getting taken down a notch legally (or in pretty much any other way) are low.
I was somewhat concerned when Google hired Kurzweil because he comes across as very Pollyanna-ish in his popular writings.
Now they're buying a company founded by the guy who created this game.
/sigh
Yet another game I could play in my Copious Free Time. I really need to figure out how to make my morning routine more efficient so I don't end up distracted by the internet when I'm lacking a hard deadline, thus recovering a few hours a day of spare time.
The predictions in his popular writings have been pretty off base. More unsettling is the way he twists the words around to pretend they're accurate.
I'm most worried about the fact that Kurzweil argued that AGI would be no threat to humans because we would "merge with the machines". He always left vague how he knew that would happen, and how he knew that would stop AI from being a threat.
Agreed, especially since, from what I’ve seen, Kurzweil’s reason for being so sanguine about Global Warming is exponential growth. He doesn’t seem to reflect on the problems that Global Warming is causing right now, or that the growth in renewables has come in a large part because of people who are concerned.
And the idea that we shouldn’t worry isn’t reassuring when it comes from someone who’s predictions of the future have mostly been incorrect. This is a man who stands by his predictions that by 2009, human musicians and cybernetic musicians would routinely play music together and that most text would come from voice recognition software, not keyboards. Anyone that takes him seriously should re-read that chapter with predictions for 2009 (which talks about 3D entertainment rooms, the growing popularity of computer authors, 3D art coming from computer artists being displayed on screens hung up on people’s houses, nanobots that think for themselves, the growing industry of creating the personalities for the artificial personas we routinely communicate with, etc.) and keep in mind that Kurzweil says his predictions were mostly accurate.