= 27d567bc2d48d9f40e9ccc20ea370b15)
yew comments on Objections to Coherent Extrapolated Volition - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (56)
It would have to turn itself off to fix the problem I am worried about. The problem is the existence of an oracle. The problem is that the first ultraintelligent machine is the last invention that man need ever make.
To fix that problem we would have to turn ourselves into superintelligences rather than creating a singleton. As long as there is a singleton that does everything that we (humanity) want, as long as we are inferior to it, all possible problems are artificially created problems that we have chosen to solve the slow way.
I believe Eliezer's discussed this issue before. If I remember correctly, he suggested that a CEV-built FAI might rearrange things slightly and then disappear rather than automatically solving all of our problems for us. Of course, we might be able to create an FAI that doesn't do that by following a different building plan, but I don't think that says anything about CEV. I'll try to find that link...
Edit: I may be looking for a comment, not a top-level post. Here's some related material, in case I can't find what I'm looking for.