= 27d567bc2d48d9f40e9ccc20ea370b15)
DanielLC comments on The idiot savant AI isn't an idiot - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (133)
That's what I mean. Since it's not quite human, the goals won't evolve quite the same way. I've seen speculation that doing nothing more than letting a human live for a few centuries would cause evolution to unagreeable goals.
A sufficiently smart AI that has sufficient understanding of its own utility function will take measures to make sure it doesn't change. If it has an implicit utility function and trusts its future self to have a better understanding of it, or if it's being stupid because it's only just smart enough to self-modify, its goals may evolve.
We know it's possible for an AI to have evolving goals because we have evolving goals.
So it's a Goldilocks AI that has stable goals :-) A too-stupid AI might change its goals without really meaning it and a too-smart AI might change its goals because it wouldn't be afraid of change (=trusts its future self).
It's not that if it's smart enough it trusts its future self. It's that if it has vaguely-defined goals in a human-like manner, it might change its goals. An AI with explicit, fully understood, goals will not change its goals regardless of how intelligent it is.