= 27d567bc2d48d9f40e9ccc20ea370b15)
XiXiDu comments on Muehlhauser-Goertzel Dialogue, Part 1 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (161)
Well, if he believes his AI will be specifically and precisely programmed so as to converge on exactly the right goals before they are solidified in the hard takeoff, then he's working on a FAI. The remaining difference in opinions would be technical - about whether his AI will indeed converge, etc. It would not be about the Scary Idea itself.
I think it's taken by Goertzel as part of the Scary Idea that it's necessary to use several orders more precise understanding of AI's goals for its behavior not to be disastrous.
It's a direct logical consequence, isn't it? If one doesn't have a precise understanding of AI's goals, then whatever goals one imparts into AI won't be precise. And they must be precise, or (step3) => disaster.
He doesn't agree that they must be precise, so I guess step 3 is also out.
He can't think that god-powerfully optimizing for a forever-fixed not-precisely-correct goal would lead to anything but disaster. Not if he ever saw a non-human optimization process at work.
So he can only think precision is not important if he believes that
(1) human values are an attractor in the goal space, and any reasonably close goals would converge there before solidifying, and/or
(2) acceptable human values form a large convex region within the goal space, and optimizing for any point within this region is correct.
Without better understanding of AI goals, both can only be an article of faith...
From the conversation with Luke, he apparently accepts faith.