= 27d567bc2d48d9f40e9ccc20ea370b15)
jimrandomh comments on [link] [poll] Future Progress in Artificial Intelligence - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (89)
The two quotes you gave say two pretty different things. What Yudkowsky said about the time-scale of self improvement being weeks or hours, is controversial. FWIW, I think he's probably right, but I wouldn't be shocked if it turned out otherwise.
What Luke said was about what happens when an already-superhuman AI gets an Internet connection. This should not be controversial at all. This is merely claiming that a "superhuman machine" is capable of doing something that regular humans already do on a fairly routine basis. The opposite claim - that the AI will not spread to everywhere on the Internet - requires us to believe that there will be a significant shift away from the status quo in computer security. Which is certainly possible, but believing the status quo will hold isn't an extreme view.
My problem with Luke's quote was the "moments later" part.
I took that as hyperbole. If I were meant to take it literally, then yes, I'd object - but I have no trouble believing that a superintelligent AI would be out of there in a matter of hours to minutes, modulo bandwidth limits, which is 'instant' enough for my purposes. Humans suck at computer security.
Yes, applying the SI definition of a moment as 1/2π seconds and the ANSI upper bound of a plural before you must change units, we can derive that he was either claiming world takeover in less than 10/(2π)^1/2 ≈ 3.9894 seconds, or speaking somewhat loosely.
Hmmmmmmmmm.
Right, and I'm saying: the "moments later" part of what Luke said is not something that should be surprising or controversial, given the premises. It does not require any thinking that can't be done in advance, which means the only limiting input is bandwidth, which is both plentiful and becoming more plentiful every year.
The premise was a superhuman intelligence? I don't see how it could create a large enough botnet, or find enough exploits, in order to be everywhere moments later. Sounds like magic to me (mind you, a complete layman).
If I approximate "superintelligence" as NSA, then I don't see how the NSA could have a trojan everywhere moments after the POTUS asked them to take over the Internet. Now I could go further and imagine the POTUS asking the NSA to take it over within 10 years in order to account for the subjective speed with which a superintelligence might think. But I strongly doubt that such a speed could make up for the data the NSA already possess and which the superintelligence still needs to acquire. It also does not make up for the thousands of drones (humans in meatspace) that the NSA controls. And since the NSA can't take over the Internet within moments I believe it is very extreme to claim that a superintelligence can. Though it might be possible within days.
I hope you don't see this as an attack. I honestly don't see how that could be possible.
Given a backdoor or an appropriate zero-day exploit, I would estimate that it would take no longer than a few minutes to gain control over most of the computers connected to the 'net if you're not worried about detection. It's not hard. Random people routinely build large botnets without any superhuman abilities.
Most computers are not directly connected to the internet.
Assuming we are not talking about computers in cars and factory control systems, this is a pretty meaningless statement. Yes, most computers sit behind routers, but then basically all computers on the 'net sit behind routers, no one is "directly" connected.
Besides, routers are computers and can be taken over as well.
But not by that zero-day microsoft exploit you found. If your router is a cisco system, you need a cisco zero-day exploit to access the machines behind it, or some other way of bypassing the firewall. Sure, it could take over all the already vulnerable computers, the same ones which are already compromised by botnets. But I object to calling these "most of the computers connected to the 'net".
The original scenario discussed was the NSA taking over the internet. I assume that the NSA has an extensive collection of backdoors and exploits (cf. Snowden) for Microsoft and Linux and Cisco, etc.
Yes well I thin XiXiDu did himself a disfavor there. If Snowden is to be believed and as various state-sponsored botnets (Stuxnet, Flame, BadBIOS(?)) have shown, the NSA has already "taken over" the internet. They may not have root access on any arbitrary internet-connected machine, but they could get it if they wanted.
My objection (and his?) is against the claim that an AI could replicate this capability in "moments," according to the "because superhuman!" line of reasoning. I find that bogus.
This is not magic, I am not a layman, and your beliefs about computer security are wildly misinformed. Putting trojans on large fractions of the computers on the internet is currently within the reach of, and is actually done by, petty criminals acting alone. While this does involve a fair amount of thinking time, all of this thinking goes into advance preparation, which could be done while still in an AI-box or in advance of an order.
Within moments? I don't take your word for this, sorry. The only possibility that comes to my mind is by somehow hacking the Windows update servers and then somehow forcefully install new "updates" without user permission.
So if I uploaded you onto some alien computer, and you had a billion years of subjective time to think about it, then within moments after you got an "Internet" connection you could put a trojan on most computers of that alien society? How would you e.g. figure out zero day exploits of software that you don't even know exists?
Well, what's going to slow it down? If you have a backdoor or an exploit, to take over a computer requires a few milliseconds for communications latency and a few milliseconds to run the code to execute the takeover. At this point the new zombie becomes a vector for further infection, you have exponential growth and BOOM!
Wouldn't have to be Windows; any popular software package with live updates would do, like Acrobat or Java or any major antivirus package. Or you could find a vulnerability that allows arbitrary code execution in any popular push notification service; find one in Apache or a comparably popular Web service, then corrupt all the servers you can find; exploit one in a popular browser, if you can suborn something like Google or Amazon's front page... there's lots of stuff you could do. If you have hours instead of moments, phishing attacks and the like become practical, and things get even more fun.
Well, presumably you're running in an environment that has some nontrivial fraction of that software floating around, or at least has access to repos with it. And there's always fuzzing.
Also, nowadays if you can suborn the cell towers taking over all the smartphones becomes fast and easy.
When you are a layman talking to experts, you should actually listen. Don't make us feel like we're wasting our time.
Care to address his valid response point in the 2nd paragraph?
Nornagest already answered it; the sets of software in and outside the box aren't disjoint.