grobstein comments on Superintelligence 10: Instrumentally convergent goals - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (31)
I wonder what kinds of insights would be available for a Superintelligence that might override it's goal-content integrity goal. To begin with, the other instrumental goals, if they emerged, would themselves tamper with its allocation of resources.
Now consider humans: sometimes we think our goals are "travel a lot" but actually they turn out to be "be on a hammock two hours a day and swim once a week".
By trial and error, we re-map our goals based on what they feel from the inside, or rewards.
Which similar processes might an AGI do?
One possible explanation for the plasticity of human goals is that the goals that change aren't really final goals.
So me-now faces the question,
If goals are interpreted widely enough, the answer should be, No. By hypothesis, those goals of me-future make no contribution to the goals of me-now, so they have no value to me. Accordingly, I should try pretty hard to prevent goal drift and / or reduce investment in the well-being of me-future.
Humans seem to answer, Yes, though. They simultaneously allow goal drift, and care about self-preservation, even though the future self may not have goals in common with the present.
This behavior can be rationalized if we assume that it's mostly instrumental goals that drift, with final goals remaining fixed. So maybe humans have the final goal of maximizing their inclusive fitness, and consciously accessible goals are just noisy instruments for this final goal. In that case, it may be rational to embrace goal drift because 1) future instrumental goals will be better suited to implementing the final goal, under changed future circumstances, and 2) allowing goals to change produces multiple independent instruments for the final goal, which may reduce statistical noise.
The first issue is that you don't know what they will be.