Comment on Fusing AI with Superstition

Wed, 21 Apr 2010 18:27:14 GMT

The problem with this idea is that it prevents us from creating an AI that is (even in principle) able to find and fix bugs in itself.

Given the size of the problem, I wouldn’t trust humans to produce a bug-free program (plus the hardware!) even after decades of code audits. So I’d very much like the AI to be capable of noticing that it has a bug. And I’m pretty sure that an AI that can figure out that it has a silly belief caused by a flipped bit somewhere will figure out why that red wire “can” transmute at a distance. If we even manage to make a hyper-intelligent machine with this kind of “superstition”, I shudder to think what might be it’s opinion on the fact that humans who built it apparently created the red wire in an attempt to manipulate the AI, thus (hyper-ironically!) sealing their fate.... — it will certainly be able to deduce all that from historical observations, recordings, etc.)

Discuss