Eliezer_Yudkowsky comments on Reply to Holden on 'Tool AI' - Less Wrong

94 Post author: Eliezer_Yudkowsky 12 June 2012 06:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (348)

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 11 July 2013 07:28:27PM 2 points [-]

That's pretty much what I mean. The point is that if you don't understand the structurally required properties well enough to describe the characteristics of a digital amp meter with a polling subroutine, saying that you'll hardwire the digital amp meter doesn't help very much. There's a hardwired version which is moderately harder to subvert on the presumption of small design errors, but first you have to be able to describe what the software does. Consider also that anything which can affect the outside environment can construct copies of itself minus hardware constraints, construct an agent that reaches back in and modifies the hardware, etc. If you can't describe how not do to this in software, 'hardwiring' won't help - the rules change somewhat when you're dealing with intelligent agents.

Comment author: bogdanb 11 July 2013 08:08:51PM 0 points [-]

the rules change somewhat when you're dealing with intelligent agents.

Now that’s an understatement!