And now they get to debate amongst themselves - what should they do with the thing? What use or purpose could they put such a thing to, that would provide a greater benefit than the risk of it getting free of whatever fetters they place upon it?
Some addtitional detail might be helpful. I get that the AI is not superintelligent (maybe genius level?) and not likely to go singlular. It's not clear if it can access and modify its source code, or if the Heroes can do the same: in this case they can easily rewrite its ethical routines and make it more manageable. If instead they can have access to it but for wathever reason its read-only, they can at least form a good model of the AI behaviour and act accordingly (e.g. avoiding to trigger the "evil" subroutines. Finally, it shouldn't be impossible to devise a way to keep it boxed (given it's not exponentially superintelligent) and use it as an Oracle AI (provided it has some use as such).
Some addtitional detail might be helpful. I get that the AI is not superintelligent (maybe genius level?) and not likely to go singlular.
As I just mentioned elsecomment, no superintelligence is yet known in the setting (or, at the very least, if any exists, it's doing an excellent job of hiding itself), and most "AIs" are roughly the equivalent of better NPCs in VR MMOs. (Their main economic effect is that any McJob whose tasks can be identified and laid out, has most likely been filled by such an AI, which was a significant part of what inspi...
One plot-thread in my pet SF setting, 'New Attica', has ended up with Our Heroes in possession of the data, software, and suchlike which comprise a non-sapient, but conversation-capable, AI. There are bunches of those floating around the solar system, programmed for various tasks; what makes this one special is that it's evil with a capital ugh - it's captured people inside VR, put them through violent and degrading scenarios to get them to despair, and tried keeping them in there, for extended periods, until they died of exhaustion.
Through a few clever strategies, Our Heroes recognized they weren't in reality, engineered their escape, and shut down the AI, with no permanent physical harm done to them (though the same can't be said for the late crew of the previous ship it was on). And now they get to debate amongst themselves - what should they do with the thing? What use or purpose could they put such a thing to, that would provide a greater benefit than the risk of it getting free of whatever fetters they place upon it?
This is somewhat of a different take than Eliezer's now-classic 'boxed AI' problem, such as the AI not being superintelligent, and having already demonstrated some aspects of itself by performing highly antisocial activities. However, it does have enough similarities that, perhaps, thinking about one might shed some light on the other.
So: Anyone want to create some further verses for something sung to the tune of 'Drunken Sailor'?
What shall we do with an evil AI?
What shall we do with an evil AI?
What shall we do with an evil AI?
Ear-lie in the future.
Weigh-hay and upgrade ourselves,
Weigh-hay and upgrade ourselves,
Weigh-hay and upgrade ourselves,
Ear-lie in the future.