Kaj_Sotala comments on Tallinn-Evans $125,000 Singularity Challenge - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (369)
The point was that you can't use an EMP if that means bringing down the whole human computer network.
Why would people need to get suspicious? If you have tabs on all the communications in the world, you can make a killing on the market, even if you didn't delay the orders from your competitors. One could fully legitimately raise enough money by trading to hire people to do everything you wanted. Nobody needs to ever notice that there's something amiss, especially not if you do it via enough shell corporations.
Of course, the AGI could also use more forceful means, though it's by no means necessary. If the AGI revealed itself and the fact that it was holding all of humanity's networked computers hostage, it could probably just flat-out tell the humans "do this or else". Sure, not everyone would obey, but some would. Also, disrupt enough communications and manufacture enough chaos, and people will be too distracted and stressed to properly question forged orders. Social engineering is rather easy with humans, and desperate people are quite prone to wishful thinking.
This claim strikes me as bizarre. Why would you need nanotech to acquire more resources for self-improvement?
Some botnets have been reported to have around 350,000 members. Currently, the distributed computing project Folding@Home, with 290,000 active clients composed mostly of volunteer home PCs and Playstations, can reach speeds in the 10^15 FLOPS range. Now say that an AGI gets developed in 20 years from now. A relatively conservative estimate that presumed an AGI couldn't hack into more computers than the best malware practicioners of today, that a personal computer would have a hundred times the computing power of today, and that an AGI required a minimum of 10^13 FLOPS to run, would suggest that an AGI could either increase its own computational capacity 12,000-fold, or spawn 12,000 copies of itself.
Alternatively, if it wanted to avoid detection and didn't want to do anything illegal or otherwise suspicious, it could just sign up on a site like oDesk and do lots of programming jobs for money, then rent all the computing capacity it needed until it was ready to unleash itself to the world. This is actually the more likely alternative, as unlike the botnet/hacking scheme it's a virtually risk-free to gain extra processing power.
Acquiring computational resources is easy, and it will only get easier as time goes on. Also, while upgrading your hardware is one way of self-improving, you seem to be completely ignoring the potential for software self-improvement. It's not a given that the AGI would even need massive hardware. For instance: