shminux comments on 'Dumb' AI observes and manipulates controllers - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (19)
I assume that you are alluding to spontaneously agentizing tools, a discussion initially triggered by the famous Karnofsky's post. In your example the feedback loop is closed, which is probably enough for a tool eventually gaining agency.
Can you link to the post you're talking about? I've been thinking about this problem, although I had never read or thought the words 'spontaneously agentizing tools.'
See Objection 2 in http://lesswrong.com/lw/cbs/thoughts_on_the_singularity_institute_si/ and related posts and comments.