You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

torekp comments on Open thread, Nov. 24 - Nov. 30, 2014 - Less Wrong Discussion

4 Post author: MrMind 24 November 2014 08:56AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (317)

You are viewing a single comment's thread. Show more comments above.

Comment author: torekp 29 November 2014 07:58:32PM 0 points [-]

I don't get this explanation. Sure, a process can tend toward a certain result, without having an explicit representation of that result. But such tendencies often seem to be fragile. For example, a car engine homeostatically tends toward a certain idle speed. But take out one or all spark plugs, and the previously stable performance evaporates. Goals-as-we-know-them, by contrast, tend to be very robust. When a human being loses a leg, they will obtain a synthetic one, or use a wheelchair. That kind of robustness is part of what makes a very powerful agent scary, because it is intimately related to the agent's seeing many things as potential resources to use toward its ends.