= 27d567bc2d48d9f40e9ccc20ea370b15)
Stuart_Armstrong comments on Siren worlds and the perils of over-optimised search - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (411)
Then if Omega is superintelligent, it has a problem: every single decision it makes increases or decreases the probability of someone answering something or other, possibly by a large amount. It seems Omega cannot avoid being coercive, just because it's so knowledgeable.
We don't quite know that, and there's also the matter of whether Omega is deliberately optimizing those people or they're just reacting to Omega's optimizing the inanimate world (which I would judge to be acceptable and, yes, unavoidable).
It is an interesting question, though, and illustrates the challenges with "liberty" as a concept in these circumstances.
Well yes. It's also why many people have argued in favor of Really Powerful Optimizers just... doing absolutely nothing.
That I don't see.