= 27d567bc2d48d9f40e9ccc20ea370b15)
Lumifer comments on [link] [poll] Future Progress in Artificial Intelligence - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (89)
Given a backdoor or an appropriate zero-day exploit, I would estimate that it would take no longer than a few minutes to gain control over most of the computers connected to the 'net if you're not worried about detection. It's not hard. Random people routinely build large botnets without any superhuman abilities.
Most computers are not directly connected to the internet.
Assuming we are not talking about computers in cars and factory control systems, this is a pretty meaningless statement. Yes, most computers sit behind routers, but then basically all computers on the 'net sit behind routers, no one is "directly" connected.
Besides, routers are computers and can be taken over as well.
But not by that zero-day microsoft exploit you found. If your router is a cisco system, you need a cisco zero-day exploit to access the machines behind it, or some other way of bypassing the firewall. Sure, it could take over all the already vulnerable computers, the same ones which are already compromised by botnets. But I object to calling these "most of the computers connected to the 'net".
The original scenario discussed was the NSA taking over the internet. I assume that the NSA has an extensive collection of backdoors and exploits (cf. Snowden) for Microsoft and Linux and Cisco, etc.
Yes well I thin XiXiDu did himself a disfavor there. If Snowden is to be believed and as various state-sponsored botnets (Stuxnet, Flame, BadBIOS(?)) have shown, the NSA has already "taken over" the internet. They may not have root access on any arbitrary internet-connected machine, but they could get it if they wanted.
My objection (and his?) is against the claim that an AI could replicate this capability in "moments," according to the "because superhuman!" line of reasoning. I find that bogus.
Let me suggest a way:
Even easier if the OS is open-sourced.
An AI probably wouldn't need to decompile anything -- given the kind of optimizations that one could apply, there's no particularly strong reason to think one would be any less comfortable in native machine code or, say, Java bytecode than in source. The only reason we are is that it's closer to natural language and we're bad at keeping track of a lot of disaggregated state.
That is a monumentally difficult undertaking, unfeasible with current hardware limitations, certainly impossible in the "moments" timescale.
I think you underestimate the state of the art, such as the SAT/SMT-solver revolution in computer security. They automatically find exploits all the time, against OSes and libraries and APIs.
I think you miss my point. These SAT solvers are extremely expensive, and don't scale well to large code bases. You can look to the literature to see the state of the art: using large clusters for long-running analysis on small code bases or isolated sections of a library. They do not and cannot with available resources scale up to large scale analysis of an entire OS or network stack ... if they did, we humans would have done that already.
So to be clear, this UFAI breakout scenario is assuming the AI already has access to massive amounts of computing hardware, which it can re-purpose to long-duration HPC applications while escaping detection. And even if you find that realistic, I still wouldn't use the word "momentarily."
Why in the world would a security audit of a bunch of code be "monumentally difficult" for an AI..?
It requires an infeasible amount of computation for us humans to do. Why do you suppose it would be different for an AI?
Doing an audit to catch all vulnerabilities is monstrously hard. But finding some vulnerabilities is a perfectly straightforward technical problem.
It happens routinely that people develop new and improved vulnerability detectors that can quickly find vulnerabilities in existing codebases. I would be unsurprised if better optimization engines in general lead to better vulnerability detectors.