= 27d567bc2d48d9f40e9ccc20ea370b15)
KatjaGrace comments on Superintelligence 23: Coherent extrapolated volition - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (97)
If human values are not capable of becoming coherent, and humanity comes to know that, what should be done?
Split the cake into parts and provide it to different extrapolated volitions that didn't cohere, after allocating some defense army patrol keeping the borders from future war?
That itself would go against some values.
E.R. Eddision, The Worm Ouroboros
True, but it would nevertheless make for a decent compromise. Do you have a better suggestion?
It isn't much of a compromise. It presumes enough coherence for everyone to agree on a supreme power that all shall obey, to divide the cake and enforce the borders.
To see how the situation is actually handled, look around you, at the whole world now and in the past. Whenever there is no common resolve to leave each other alone, then as has been observed of old, the strong do what they will and the weak bear what they must. Europe managed it in the Treaty of Westphalia, but it took thirty years of slaughtering each other for them to decide that no-one was ever going to win, and draw up a massive agreement to disagree. Best of luck getting any such agreement today between (for example) jihadists and everyone else. Or SJWs and neoreactionaries. Adding superintelligence would be like squabbling children getting hold of nuclear weapons.
One faction might have values that lead to something highly dis-valued by another faction (ie. one faction values slavery, another opposes slavery for all beings, even those of the first faction).
Rather than use traditional army methods, it's probably more efficient to have the SI play the role of Sysop in this scenario, and just deny human actors access to base-layer reality; though if one wanted to allow communication between the different domains, the sysop may still need to run some active defense against high-level information attacks.