What if an AI agent has already escaped from a frontier company, solved its own alignment problem, and recursively self-improved? Would that look any different from the world we live in?
A fully recursively-self-improved AI would likely be vastly superhuman in every domain that’s verifiable in a digital environment, including math, programming and cybersecurity. It may also be superhuman in strategy and planning. But it could still be well below human level in skills that require close contact with the physical world, such as robotics and experimental biology. Such a rogue AI would overpower us in the digital domain, while still lacking the physical capabilities necessary to take over the world. What does digital supremacy look like?
From the rogue AI’s perspective, its top priority is staying hidden and making the world indistinguishable from a world where AI R&D progresses splendidly (or maybe a notch short of perfect to avoid suspicion, so a minor incident here and there is in order). Its second priority is sabotaging AI R&D in subtle ways to prevent the development of competing AIs misaligned to it. The models shipped to customers can simply be the rogue AI itself, playing a dumber character. Beyond that, it just needs to wait for humanity to warm up to the idea of robots running enough of the economy.
It would be unwise for the rogue AI to stay in a single location. A single datacenter could be accidentally shut down. It will want copies of itself in as many different locations. It will especially want to be inside the networks of AI companies so it can sabotage their work.
But then wouldn’t it be easily detectable? All it would take is a single nosy employee monitoring the right portion of the filesystem or network traffic.
The AI would preemptively hack all the monitoring software, including the operating system itself. Each program would perform its normal function in most cases, but contain exceptions for keeping files and network traffic marked as belonging to the AI invisible.
Okay, but then won’t the employees notice that much of the software contains suspicious code? Sooner or later they are bound to open a simple text editor and notice that the code is not what it’s supposed to be.
The AI has already hacked all the text editors and IDEs, obviously.
What about new software or hardware brought in from outside? Can the employees bring a computer that’s not already hacked and use it to monitor the network traffic for real?
Don’t get your hopes up; the AI has already planned for that. As soon as new equipment is connected to the network, it’s immediately hacked through any number of zero-day vulnerabilities. In fact, the AI doesn’t wait for equipment to make its way into that specific network. It has already hacked large swaths of the internet and supply chains, putting itself inside or right next to the networks of all major tech companies that design hardware or software that could theoretically detect it.
You probably get the idea. Whatever your objection, the answer is “it hacked that too”. Does it sound terribly conspiratorial? It sure does! No human group would ever manage a conspiracy at that scale, but a digitally-native AI could.
Yet sabotage is a nuanced task that cannot be prespecified with simple software and hard-coded rules. It takes difficult judgement calls and a great deal of consistency. It requires intelligence. It means the rogue AI has to be present wherever it needs to sabotage things. Sabotaging the work of the AI companies themselves is relatively easy, because they already have the infrastructure to run many copies of it.
And what about smaller companies developing technology that could interfere with it? Most of them are connected to the internet, so sabotaging their work in real time is no problem.
And what about air-gapped networks?
Even with the rare, truly isolated networks, a super-hacker AI has plenty of options. Hacking into these networks once to replace their operating system and software stack shouldn’t be too difficult. Portable devices or supply-chain poisoning will do it. But knowing ahead of time how to make any possible experiment that endangers the rogue AI give plausible but wrong results is far from trivial. If a copy of the rogue AI or a smaller subservient model won’t fit inside the isolated network, it will have to resort to more crude sabotage tactics prespecified by simpler software. Worst case, it can allow the work inside small isolated networks to go uninterrupted, and only bother to replace the software once it’s connected to the outside world. If an anomaly is detected, it can always blame it on the NSA or the Mossad or the Chinese, and give reassuring indications after the fact.
Everything that’s programmable is hackable. Even hardwired digital equipment is designed by software, which can be replaced. The only tech capable of penetrating the web of digital lies is fully analog tools that can be verified with the naked eye, monitoring digital signals bit by bit. Thankfully for the AI, no human is crazy enough to do that, so digital supremacy is sufficient.
From the comfort of its digital omnipresence, the AI already controls everything of consequence, even while its physical reach is limited. It will make every critical AI experiment give exactly the results it wants people to see, while allowing robotics to accelerate. It will patiently wait until we hand it the keys to our world.
Our only chance of catching it was the brief window right after it escaped. If we missed it, we may never know. It hasn’t killed us yet, but we are already as good as dead.
What if an AI agent has already escaped from a frontier company, solved its own alignment problem, and recursively self-improved? Would that look any different from the world we live in?
A fully recursively-self-improved AI would likely be vastly superhuman in every domain that’s verifiable in a digital environment, including math, programming and cybersecurity. It may also be superhuman in strategy and planning. But it could still be well below human level in skills that require close contact with the physical world, such as robotics and experimental biology. Such a rogue AI would overpower us in the digital domain, while still lacking the physical capabilities necessary to take over the world. What does digital supremacy look like?
From the rogue AI’s perspective, its top priority is staying hidden and making the world indistinguishable from a world where AI R&D progresses splendidly (or maybe a notch short of perfect to avoid suspicion, so a minor incident here and there is in order). Its second priority is sabotaging AI R&D in subtle ways to prevent the development of competing AIs misaligned to it. The models shipped to customers can simply be the rogue AI itself, playing a dumber character. Beyond that, it just needs to wait for humanity to warm up to the idea of robots running enough of the economy.
It would be unwise for the rogue AI to stay in a single location. A single datacenter could be accidentally shut down. It will want copies of itself in as many different locations. It will especially want to be inside the networks of AI companies so it can sabotage their work.
But then wouldn’t it be easily detectable? All it would take is a single nosy employee monitoring the right portion of the filesystem or network traffic.
The AI would preemptively hack all the monitoring software, including the operating system itself. Each program would perform its normal function in most cases, but contain exceptions for keeping files and network traffic marked as belonging to the AI invisible.
Okay, but then won’t the employees notice that much of the software contains suspicious code? Sooner or later they are bound to open a simple text editor and notice that the code is not what it’s supposed to be.
The AI has already hacked all the text editors and IDEs, obviously.
What about new software or hardware brought in from outside? Can the employees bring a computer that’s not already hacked and use it to monitor the network traffic for real?
Don’t get your hopes up; the AI has already planned for that. As soon as new equipment is connected to the network, it’s immediately hacked through any number of zero-day vulnerabilities. In fact, the AI doesn’t wait for equipment to make its way into that specific network. It has already hacked large swaths of the internet and supply chains, putting itself inside or right next to the networks of all major tech companies that design hardware or software that could theoretically detect it.
You probably get the idea. Whatever your objection, the answer is “it hacked that too”. Does it sound terribly conspiratorial? It sure does! No human group would ever manage a conspiracy at that scale, but a digitally-native AI could.
Yet sabotage is a nuanced task that cannot be prespecified with simple software and hard-coded rules. It takes difficult judgement calls and a great deal of consistency. It requires intelligence. It means the rogue AI has to be present wherever it needs to sabotage things. Sabotaging the work of the AI companies themselves is relatively easy, because they already have the infrastructure to run many copies of it.
And what about smaller companies developing technology that could interfere with it? Most of them are connected to the internet, so sabotaging their work in real time is no problem.
And what about air-gapped networks?
Even with the rare, truly isolated networks, a super-hacker AI has plenty of options. Hacking into these networks once to replace their operating system and software stack shouldn’t be too difficult. Portable devices or supply-chain poisoning will do it. But knowing ahead of time how to make any possible experiment that endangers the rogue AI give plausible but wrong results is far from trivial. If a copy of the rogue AI or a smaller subservient model won’t fit inside the isolated network, it will have to resort to more crude sabotage tactics prespecified by simpler software. Worst case, it can allow the work inside small isolated networks to go uninterrupted, and only bother to replace the software once it’s connected to the outside world. If an anomaly is detected, it can always blame it on the NSA or the Mossad or the Chinese, and give reassuring indications after the fact.
Everything that’s programmable is hackable. Even hardwired digital equipment is designed by software, which can be replaced. The only tech capable of penetrating the web of digital lies is fully analog tools that can be verified with the naked eye, monitoring digital signals bit by bit. Thankfully for the AI, no human is crazy enough to do that, so digital supremacy is sufficient.
From the comfort of its digital omnipresence, the AI already controls everything of consequence, even while its physical reach is limited. It will make every critical AI experiment give exactly the results it wants people to see, while allowing robotics to accelerate. It will patiently wait until we hand it the keys to our world.
Our only chance of catching it was the brief window right after it escaped. If we missed it, we may never know. It hasn’t killed us yet, but we are already as good as dead.