You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

leplen comments on Superintelligence Reading Group - Section 1: Past Developments and Present Capabilities - Less Wrong Discussion

25 Post author: KatjaGrace 16 September 2014 01:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (232)

You are viewing a single comment's thread. Show more comments above.

Comment author: leplen 18 September 2014 11:15:24PM 1 point [-]

While I agree that it is out of control if the behavior is neither ordered nor wanted, I think it's also very possible for the system to get out of control while doing exactly what you ordered it to, but not what you meant for it to.

The argument I'm making is approximately the same as the one we see in the outcome pump example.

This is to say, while a system that is doing something neither ordered nor wanted is definitely out of control, it does not follow that a system that is doing exactly what it was ordered to do is necessarily under your control.