gattsuru comments on The Hidden Complexity of Wishes - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (121)
Even what you really mean may not be what you should be wishing for, if you don't have complete information, but that's honestly the least of the relevant problems. We've got a hell of a time just getting computers to understand human speech : it's taken decades to achieve the idiot-listeners on telephone lines. By the point where you can point an AGI at yourself and tell it to do what I mean, you've either programmed it with a non-trivial set of human morality or taught it to program itself with a non-trivial portion of human morality.
You might as well skip the wasted breath and opaqueness. That's a genie that's safe enough to simply ask to do as you should wish, aka Friendly-AI-complete.
((On top of /that/, the more complex the utility function, the more likely you are to get killed by value drift down the road, when some special-case patch or rule doesn't correctly transfer from your starting FAI to its next generation, and eventually you end up with a very unfriendly AI, or when the scales get large enough that your initial premises no longer survive.))