Assuming the possibility of all of the following: what would happen if every person had a superintelligent AI with a utility function of that person's idealized extrapolated utility function? How would that compare to a scenario with a single AI embodying a successful calculation of CEV?
A singleton AI with individual CEV's for each human can do at least as well by simulating the negotiation of uniformly powerful individual AIs for each CEV. This is more stable by having the singleton's simulation enforce uniform levels of power, where actual AIs could potentially diverge in power.
I don't think "individual CEV" is proper. It's like calling an ATM an "ATM organism", which would be even worse than calling it an "ATM machine", as is common. The "C" means individual extrapolated volitions are combined coherently.
I agree it would in theory be better to have a singleton. But that requires knowing how to cohere extrapolated volitions. My idea is that it might be possible to push off that task to superintelligences without destroying the world in the process.
Questions for discussion, with my tentative answers. Assuming I am wrong about some things, there is something interesting to consider. This is inspired by the recent SL4-type and CEV-centric topics in the discussion section.
Questions:
I
II
III
My Answers:
I
II
III