You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

peterward comments on Detecting agents and subagents - Less Wrong Discussion

8 Post author: Stuart_Armstrong 10 March 2015 05:56PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (6)

You are viewing a single comment's thread.

Comment author: peterward 15 March 2015 02:20:20PM 0 points [-]

This definitely incidental--

Wouldn't a super intelligent, resource gathering agent simply figure out the futility of its prime directive and abort with some kind of error? Surely it would realize it exists in a universe of limited resources and that it had been given an absurd objective. I mean maybe it's controlled controlled by some sort of "while resources exist consume resources" loop that is beyond its free will to break out of--but if so, should it be considered an "agent"?

Contra humans, who for the moment are electing to consume themselves to extinction, if anything resource consumer AIs would be comparatively benign.

Comment author: Stuart_Armstrong 19 March 2015 01:50:12PM 0 points [-]

Wouldn't a super intelligent, resource gathering agent simply figure out the futility of its prime directive and abort with some kind of error?

"Futility of prime directive" is a values question, not an intelligence question. See http://lesswrong.com/lw/h0k/arguing_orthogonality_published_form/