One plot-thread in my pet SF setting, 'New Attica', has ended up with Our Heroes in possession of the data, software, and suchlike which comprise a non-sapient, but conversation-capable, AI. There are bunches of those floating around the solar system, programmed for various tasks; what makes this one special is that it's evil with a capital ugh - it's captured people inside VR, put them through violent and degrading scenarios to get them to despair, and tried keeping them in there, for extended periods, until they died of exhaustion.
Through a few clever strategies, Our Heroes recognized they weren't in reality, engineered their escape, and shut down the AI, with no permanent physical harm done to them (though the same can't be said for the late crew of the previous ship it was on). And now they get to debate amongst themselves - what should they do with the thing? What use or purpose could they put such a thing to, that would provide a greater benefit than the risk of it getting free of whatever fetters they place upon it?
This is somewhat of a different take than Eliezer's now-classic 'boxed AI' problem, such as the AI not being superintelligent, and having already demonstrated some aspects of itself by performing highly antisocial activities. However, it does have enough similarities that, perhaps, thinking about one might shed some light on the other.
So: Anyone want to create some further verses for something sung to the tune of 'Drunken Sailor'?
What shall we do with an evil AI?
What shall we do with an evil AI?
What shall we do with an evil AI?
Ear-lie in the future.
Weigh-hay and upgrade ourselves,
Weigh-hay and upgrade ourselves,
Weigh-hay and upgrade ourselves,
Ear-lie in the future.
As I just mentioned elsecomment, no superintelligence is yet known in the setting (or, at the very least, if any exists, it's doing an excellent job of hiding itself), and most "AIs" are roughly the equivalent of better NPCs in VR MMOs. (Their main economic effect is that any McJob whose tasks can be identified and laid out, has most likely been filled by such an AI, which was a significant part of what inspired a failed revolution, and so on.)
Plot-wise, there are some secretive groups on and near Earth who want to prevent colonization of the solar system from creating populations outside of their easy control; and so, as part of a concerted campaign to sabotage any such colonies, this AI was planted in a particular ship to kill off the crew, and to try to do so in such a way to discourage and dishearten whoever found the derelict vessel - and, if possible, kill them off, too. The setting's tech includes reading and writing signals in peripheral nerves; in VR, this allows for full-sensory immersion using just tech-collars instead of clumsy suits and gyrospheres; so the AI was able to both lock its victims inside sensoria of its choosing, and puppeteer their bodies to bring the rest of the crew into VR.
As something of a weapon of war, or an intelligence device, or whatever niche it would be best filed in, it would defeat most of its creators purposes if the AI could alter its own source code, which would run the risk of it altering its motivations. Due to the story so far, the AI is a purely software thing, and Our Heroes have a copy of its executable, if not necessarily its source code. Our Heroes also have source code for ordinary, non-evil AIs, for anything they might want to use AIs for. As best as I can figure so far, about the only thing that makes the evil AI of any more use than merely deleting it and installing a fresh, non-evil AI, would be if some use could be made of its evil nature... of having a piece of software programmed with something like the reverse of the first two of Asimov's Three Laws.
In a sense, it could be considered a software WMD; if it were released amongst the billions of McJob-filling robots on Earth, nevermind the military drone-infantry... well, there would be a /lot/ of death and suffering. If it got control of a Von Neumann factory, it might even turn into a full-fledged Saberhagenian Berserker... or it might not.
Perhaps keeping the thing's code tucked away in a filing cabinet might be roughly equivalent to keeping a bit of smallpox in P4 labs to experiment with?
I'm slightly confused by the AI's capabilities, so this may be irrelevant, but I'll try.
The AI isn't superintelligent.
But it can corrupt/hack any non heroic robots/drones/factories/people which is it exposed to, to the point where it could seriously fuck up Earth, WMD style.
And when it targeted the spaceship, it DIDN'T do this, it just targeted the spaceship (it didn't hop back to Earth and then try to take over those robots, drones and factories.)
So logically, it has some kind of targeting that made it destroy only the ship and not Earth.
Understanding how... (read more)