Ultimate control and getting defeated don't mesh well. In Hollywood there a chance that an AGI that gets ultimate control gets afterwards defeated. In the real world not so much.
How you do measure, besides your gut feelings, realisticness of these kinds of scenarios?
Analysis in multiple different ways. Keeping up with the the discourse.
There is no way to assign probabilities accurately
That's debatable. You can always use Bayesian reasoning but it's not the main issue of this debate.
Oh thanks, now I see it, these almost-there cases looking somewhat holywoodish, like the villain obligatory should pronounce lingering monologue before actually killing his victim, and thanks God hero appears in last moment.
Okay, Skynet which won't instantly get rid of humanity is improbable, if really superintelligent, and if it has this goal.
We can easily imagine many kinds of possible catastrophe which can happen, but we are not equally good at producing heavenlike utopian views, but this is an evidence only about our lack of imagination
I've been returning to my "reduced impact AI" approach, and currently working on some idea.
What I need is some ideas on features that might distinguish between an excellent FAI outcome, and a disaster. The more abstract and general the ideas, the better. Anyone got some suggestions? Don't worry about quality at this point, originality is more prized!
I'm looking for something generic that is easy to measure. At a crude level, if the only options were "papercliper" vs FAI, then we could distinguish those worlds by counting steel content.
So basically some more or less objective measure that has a higher proportion of good outcomes than the baseline.