Two problems: An obnoxious optimizing process isn't necessarily sentient. And how much would you really want such a continuation if it say tried to put everything in its future lightcone into little smiley faces?
If it helps ask yourself how you feel about a human empire that expands through its lightcone preemptively destroying every single alien species before they can do anything with a motto of "In the Prisoners' Dilemma, Humanity Defects!" That sounds pretty bad doesn't it? Now note that the AGI expansion is probably worse than that.
Two problems: An obnoxious optimizing process isn't necessarily sentient.
Hence my caveat.
And how much would you really want such a continuation if it say tried to put everything in its future lightcone into little smiley faces?
I find the plausibility of a sentient AGI constrained to such a value to be vanishingly small.
If it helps ask yourself how you feel about human empire that expands through its lightcone preemptively destroying every single alien species before they can do anything with a motto of "In the Prisoner's Dilemma, Humanity Defects!" That sounds pretty bad doesn't it?
Not especially, no.
Similar to the monthly Rationality Quotes threads, this is a thread for memorable quotes about Artificial General Intelligence.