Eugine_Nier comments on Ethical dilemmas for paperclip maximizers - Less Wrong

11 Post author: CronoDAS 01 August 2011 05:31AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (24)

You are viewing a single comment's thread. Show more comments above.

Comment author: Eugine_Nier 02 August 2011 07:14:57AM 0 points [-]

It is possible that even the specification of a paperclip is unstable under recursive improvement but somewhat less likely. Postulating agents that don't even know what a paperclip is seems less useful as a tool for constructing counterfactuals.

It is, however, useful for thinking about recursive stability in general, and thinking about designing agents to have stable goal systems.