warbo comments on LessWrong's attitude towards AI research - Less Wrong

8 Post author: Florian_Dietz 20 September 2014 03:02PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (49)

You are viewing a single comment's thread. Show more comments above.

Comment author: warbo 22 September 2014 12:13:17PM 1 point [-]

By an off switch I mean a backup goal. Goals are standardly regarded as immune self modification, so an off switch, in my sense, would be too.

This is quite a subtle issue.

If the "backup goal" is always in effect, eg. it is just another clause of the main goal. For example, "maximise paperclips" with a backup goal of "do what you are told" is the same as having the main goal "maximise paperclips while doing what you are told".

If the "backup goal" is a separate mode which we can switch an AI into, eg. "stop all external interaction", then it will necessarily conflict with the the AI's main goal: it can't maximise paperclips if it stops all external interaction. Hence the primary goal induces a secondary goal: "in order to maximise paperclips, I should prevent anyone switching me to my backup goal". These kind of secondary goals have been raised by Steve Omohundro.

Comment author: TheAncientGeek 22 September 2014 12:46:22PM 1 point [-]

You haven't dealt with the case where the safety goals are the primary ones.

These kinds of primary goals have been raised by Isaac Asimov.

Comment author: FeepingCreature 22 September 2014 04:13:52PM *  1 point [-]

The question of "what are the right safety goals" is what FAI research is all about.