All of kitpeddler's Comments + Replies

The keywords for this concern are s-risk / astronomical suffering. I think this is unlikely, since a wrapper-mind that would pursue suffering thereby cares about human-specific concerns, which requires alignment-structure. So more likely this either isn't systematically pursued (other than as incidental mindcrime, which allows suffering but doesn't optimize for it), or we get full alignment (for whatever reason).

One line of reasoning is as follows: 1. We don't know what goal(s) the AGI will ultimately have.   (We can't reliably ensure what those goals are.) 2. There is no particular reason to believe it will have any particular goal.   3. Looking at all the possible goals that it might have, goals of explicitly benefiting or harming human beings are not particularly likely. 4. On the other hand, because human beings use resources which the AGI might want to use for its own goals and/or might pose a threat to the AGI (by, e.g. creating other AGIs) there are reasons why an AGI not dedicated to harming or benefiting humanity might destroy humanity anyway. (This is an example or corollary of  "instrumental convergence".) 5. Because of 3, minds tortured for eternity is highly unlikely.   6. Because of 4, humanity being ended in the service of some alien goal which has zero utility from the perspective of humanity is far more likely.