You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Nornagest comments on Everybody's talking about machine ethics - Less Wrong Discussion

14 Post author: sbenthall 17 September 2014 05:20PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (16)

You are viewing a single comment's thread. Show more comments above.

Comment author: Nornagest 22 September 2014 09:39:23PM *  3 points [-]

Presumably the reason there is no mention of superintelligence or recursively self-modifying anything because those concepts don't exist with today's technology.

Superintelligence doesn't, but recursive self-modification has been a feature of AI research since the Seventies. As MIRI predicts, value stability proved to be a problem.

(Eurisko's authors solved it -- allegedly, there isn't much open data -- by walling off the agent's utility function from modification. This would be much harder to do to a superintelligent agent.)