= 27d567bc2d48d9f40e9ccc20ea370b15)
TheAncientGeek comments on Debunking Fallacies in the Theory of AI Motivation - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (343)
The only sense in which the "rigidity" of goals can be said to be a universal fact about minds is that it is these goals that determine how the AI will modify itself once it has become smart and capable enough to do so. It's not a good idea to modify your goals if you want them to become reality; that seems obviously true to me, except perhaps for a small number of edge cases related to internally incoherent goals.
Your points against the inevitability of goal rigidity don't seem relevant to this.
If you take the binary view that you're either smart enough to achieve your goals or not, then you might well want to stop improving when you have the minimum intelligence necessary to meet them...which means, among other things,that AIs with goals requiring human or lower intelligence won't become superhuman .... which lowers the probability of the Clippie scenario. It doesn't require huge intelligence to make paperclips,so an AI with a goal to make paperclips, but not to make any specific amount, wouldn't grow into a threatening monster.
The probability of the Clippie scenario is also lowered by the consideration that fine grained goals might shift during self-improvement phase, so the Clippie scenario .... arbitrary goals combined with a superintelligence .... is whittled away from both ends.