Viliam_Bur comments on Rationality Quotes October 2013 - Less Wrong

7 [deleted] 05 October 2013 09:02PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (313)

You are viewing a single comment's thread. Show more comments above.

Comment author: Viliam_Bur 16 October 2013 12:03:28PM 4 points [-]

goal system that is so incomprehensible to humans as to be indistinguishable from no goal system at all

No, "incomprehensible" doesn't imply "invisible". That's like saying that impersonal laws of physics are indistinguishable from having no laws of physics at all. Gravity is impersonal, yet it is personally observable.

In a similar way, we could observe an appearance of mountains of paperclips, even if we had no clue about why the AGI is doing that (assuming we would still exist and had an access to the AGI's code and data).

we would not notice a superintelligence if it were staring us in the face, provided it decided to keep humans around

Provided it decided to keep humans around in their original environment. Otherwise we would notice a change in the environment, even if we couldn't discover its cause.