= 27d567bc2d48d9f40e9ccc20ea370b15)
dspeyer comments on The Goal of the Bayesian Conspiracy - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (80)
This needs a safety hatch.
It is a recurring pattern in history for determined, well-intentioned people to seize power and then do damage. Certainly we're different because we're rational, but they were different because they were ${virtueTheyValueMost}. See also The Outside View and The Sorting Hat's Warning.
A conspiracy of rationalists is even more disturbing because of how closely it resembles an AI. As individuals, we balance more logic based on our admittedly underspecified terminal values against moral intuition. But our intuitions do not match, nor do we communicate them easily. So collectively moral logic dominates. Pure moral logic without really good terminal values... we've been over this.
Don't worry. This is exactly what the Contrarian Conspiracy was designed to prevent.
Everything is going according to plan.
Huh. An interesting point, and one that I should have considered. So what would you suggest as a safety hatch?
I don't know, but I'll throw some ideas up. These aren't all the possibilities and probably don't include the best possibility.
Each step must be moral taken in isolation. No it'll-be-worth-it-in-ten-years reasoning, since that can go especially horribly wrong.
Work honestly within the existing systems. This allows existing safeguards to apply. On the other hand, it assumes it's possible to get anything done within existing systems by being honest.
Establish some mechanism to keep moral intuition. Secret-ballot mandatory does-this-feel-right votes.
Divide into several conspiracies, which are forbidden to have discuss issues with eachother, preventing groupthink.
Have an oversight conspiracy, with the power to shut us down if they believe we've gone evil.