a bunch of ideas! [edit: oops, I answered positive-side only.] [edit #2: I liked another answer so much I strong downvoted myself to put theirs higher]
General success:
General failures:
First off, I dislike the FOOM story here, since it implies that AI risk is only based on FOOM.
But there are multiple paths to success, which is why I'm surprisingly optimistic here.
But the biggest paths are:
Are there any humans still around?
Has deceptive alignment been solved?
What counts as a human? How would you know from outside if "deceptive alignment" has been solved?
Suppose you are put in stasis and wake up 10 years after the FOOM. You are trying to figure out if the AI Alignment project succeeded or not. How can you tell? Not vaguely, concretely. What metrics indicate success, and what metrics indicate failure? The following are potential examples, based on various sci-fi AI tropes. No need to discuss each one separately, there are millions more, the idea is to delineate failures from success in the general case.
Any links discussing this would be appreciated, too.