1. All a self-improving AI needs is good enough YGBM (you gotta believe me) technology. Eliezer gets this, some of the commenters in this thread don't. The key isn't the ability to solve protein-folding, the key is the ability to manipulate a threshhold amount of us to its ends. 2. We may already functionally be there. You and Chiquita Brands International in a death match. My money is CBI killing you before you kill it. Companies and markets already manipulate our behavior, using reward incentives like the ones in Eliezer's parable for us to engage in behavior that maximizes their persistence at the expense of ours. Whether they're "intelligent", "conscious", or not isn't as relevant to me as the fact that they may have a permanent persistence maximizing advantage over us. My money is probably on corporations and markets substrate jumping and leaving us behind as more likely than us substrate jumping and leaving our cellular and bacteriological medium behind.