= 27d567bc2d48d9f40e9ccc20ea370b15)
Will_Newsome comments on Evidence for the orthogonality thesis - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (289)
How is this a curiosity stopper? It's a good question, as is evidenced by your trying to find an answer to it.
It's a curiosity stopper in the sense that people don't worry any more about risks from AI when they assume that intelligence correlates with doing the right thing, and that superintelligence would do the right thing all the time.
Stuart is trying to answer a different question, which is "Given that we think that's probably false, what are some good examples that help people to see its falsity?"
"What should we have the AI's goals be?"
"Eh, just make it self-improve, once it's smart it can figure out the right goals."
How's that any different from:
"What should we have the AI's beliefs be?"
"Eh, just make it self-improve, once it's smart it can figure out the true beliefs."
It's not very different. They are both different from:
"The AI will acquire accurate beliefs by using a well understood epistemology to process its observations, as it is explicitly designed to do."
"Smart" implicitly entails "knows the true beliefs", whereas it doesn't entail "has the right goals".
It doesn't exclude having the right goals, either. You could engineer something whose self-improvement was restricted from affecting its goals. But if that is dangerous, why would you?
Well, the difference is that building an AI without figuring out where goals come from gives you a dangerous AI, while building an AI without figuring out where beliefs come from gives you a highly-optimized compiler that wants to save humanity.
factual beliefs != moral beliefs
And the methods for investigating them are very different.