army1987 comments on Isolated AI with no chat whatsoever - Less Wrong

14 Post author: ancientcampus 28 January 2013 08:22PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (61)

You are viewing a single comment's thread.

Comment author: [deleted] 29 January 2013 10:55:22PM 1 point [-]

Suppose you make a super-intelligent AI and run it on a computer. The computer has NO conventional means of output (no connections to other computers, no screen, etc).

Why would anyone do that?

Comment author: ewbrownv 30 January 2013 05:39:32PM 1 point [-]

Indeed. What's the point of building an AI you're never going to communicate with?

Also, you can't build it that way. Programs never work the first time, so at a minimum you're going to have a long period of time where programmers are coding, testing and debugging various parts of the AI. As it nears completion that's going to involve a great deal of unsupervised interaction with a partially-functional AI, because without interaction you can't tell if it works.

So what are you going to do? Wait until the AI is feature-complete on day X, and then box it? Do you really think the AI was safe on day X-1, when it just had a couple of little bugs left? How about on day X-14, when you thought the major systems were all working but there was actually a major bug in the expected utility calculator? Or on day X-60, when a programmer got the Bayesian reasoning system working but it was connected to a stubbed-out version of the goal system instead of the real thing?

This myopic focus on boxing ideas misses most of the problems inherent in building a safe AGI.