Thanks for this analysis and congratulations on its clarity.
One important point is that that when considering:
π5: Shut down and let the humans optimize whatever V they have in the actual world
the text makes it look like the outcome involves unassisted humans optimising V. But the real option is that humans would almost certainly be trying to optimise V with future AI assistance, quite possibly by an improved, restarted version of the AI itself. I think this helps make it a lot more plausible that this is the best policy for the AI to choose (both intuitiv... (read more)
Thanks for this analysis and congratulations on its clarity.
One important point is that that when considering:
π5: Shut down and let the humans optimize whatever V they have in the actual world
the text makes it look like the outcome involves unassisted humans optimising V. But the real option is that humans would almost certainly be trying to optimise V with future AI assistance, quite possibly by an improved, restarted version of the AI itself. I think this helps make it a lot more plausible that this is the best policy for the AI to choose (both intuitiv... (read more)