grobstein comments on Superintelligence 10: Instrumentally convergent goals - Less Wrong

7 Post author: KatjaGrace 18 November 2014 02:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (31)

You are viewing a single comment's thread. Show more comments above.

Comment author: grobstein 18 November 2014 07:44:43PM 1 point [-]

One possible explanation for the plasticity of human goals is that the goals that change aren't really final goals.

So me-now faces the question,

Should I assign any value to final goals that I don't have now, but that me-future will have because of goal drift?

If goals are interpreted widely enough, the answer should be, No. By hypothesis, those goals of me-future make no contribution to the goals of me-now, so they have no value to me. Accordingly, I should try pretty hard to prevent goal drift and / or reduce investment in the well-being of me-future.

Humans seem to answer, Yes, though. They simultaneously allow goal drift, and care about self-preservation, even though the future self may not have goals in common with the present.

This behavior can be rationalized if we assume that it's mostly instrumental goals that drift, with final goals remaining fixed. So maybe humans have the final goal of maximizing their inclusive fitness, and consciously accessible goals are just noisy instruments for this final goal. In that case, it may be rational to embrace goal drift because 1) future instrumental goals will be better suited to implementing the final goal, under changed future circumstances, and 2) allowing goals to change produces multiple independent instruments for the final goal, which may reduce statistical noise.

Comment author: Lumifer 18 November 2014 07:48:53PM 1 point [-]

Should I assign any value to final goals that I don't have now, but that me-future will have because of goal drift?

The first issue is that you don't know what they will be.