DanArmak comments on Why the beliefs/values dichotomy? - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (153)
(Edited & corrected) Here's a third example. Imagine an AI whose only supergoal is to gather information about something. It explicitly encodes this information, and everything else it knows, as a Bayesian network of beliefs. Its utility ultimately derives entirely from creating new (correct) beliefs.
This AI's values and beliefs don't seem very separate to me. Every belief can be mapped to the value of having that belief. Values can be mapped to the belief(s) from whose creation or updating they derive. Every change in belief corresponds to a change in the AI's current utility, and vice versa. Given a subroutine fully implementing the AI's belief subsystem, the value system would be relatively simple, and vice versa.
However, this doesn't imply the AI is in any sense simple or incapable of adaptation. Nor should it imply (though I'm no AI expert) that the AI is not a 'mind' or is not conscious. Similarly, while it's true that the Roomba doesn't have a belief/value separation, that's not related to the fact that it's a simple and stupid 'mind'.