Comment on Rationality Quotes October 2013

Wed, 16 Oct 2013 12:03:28 GMT

goal system that is so incomprehensible to humans as to be indistinguishable from no goal system at all

No, "incomprehensible" doesn't imply "invisible". That's like saying that impersonal laws of physics are indistinguishable from having no laws of physics at all. Gravity is impersonal, yet it is personally observable.

In a similar way, we could observe an appearance of mountains of paperclips, even if we had no clue about why the AGI is doing that (assuming we would still exist and had an access to the AGI's code and data).

we would not notice a superintelligence if it were staring us in the face, provided it decided to keep humans around

Provided it decided to keep humans around in their original environment. Otherwise we would notice a change in the environment, even if we couldn't discover its cause.

Discuss