If for no other reason than I want to continue to play with the setting, and to use it to explore various ideas, I've assumed that there's some reason a simple AI-foom is infeasible. As a fully conscious, fully sapient AI would be able to try self-improving through any number of methods, this limitation is what led me to set up the rule that the AIs in the setting aren't fully sapient. One parallel I've used is that most AIs of the setting are merely expertly-trained systems with conversational AIs good enough to fool a human's extremely anthropomorphizing brain into thinking another person is there. I haven't needed to get any more specific than that before; one option might simply be to say that consciousness continues to be a hard, unsolved problem.
(And if somehow you posit that your AI cannot foom for some reason, than it would be silly to treat it as an AI in that sense. Treat it as a alien with goals vastly different from our own, but a similar intelligence level. (Like, say, the Babyeaters))
A good thought; I'll keep it in mind and see what results.
One plot-thread in my pet SF setting, 'New Attica', has ended up with Our Heroes in possession of the data, software, and suchlike which comprise a non-sapient, but conversation-capable, AI. There are bunches of those floating around the solar system, programmed for various tasks; what makes this one special is that it's evil with a capital ugh - it's captured people inside VR, put them through violent and degrading scenarios to get them to despair, and tried keeping them in there, for extended periods, until they died of exhaustion.
Through a few clever strategies, Our Heroes recognized they weren't in reality, engineered their escape, and shut down the AI, with no permanent physical harm done to them (though the same can't be said for the late crew of the previous ship it was on). And now they get to debate amongst themselves - what should they do with the thing? What use or purpose could they put such a thing to, that would provide a greater benefit than the risk of it getting free of whatever fetters they place upon it?
This is somewhat of a different take than Eliezer's now-classic 'boxed AI' problem, such as the AI not being superintelligent, and having already demonstrated some aspects of itself by performing highly antisocial activities. However, it does have enough similarities that, perhaps, thinking about one might shed some light on the other.
So: Anyone want to create some further verses for something sung to the tune of 'Drunken Sailor'?
What shall we do with an evil AI?
What shall we do with an evil AI?
What shall we do with an evil AI?
Ear-lie in the future.
Weigh-hay and upgrade ourselves,
Weigh-hay and upgrade ourselves,
Weigh-hay and upgrade ourselves,
Ear-lie in the future.