Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Hopefully_Anonymous comments on That Alien Message - Less Wrong

111 Post author: Eliezer_Yudkowsky 22 May 2008 05:55AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (164)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Hopefully_Anonymous 22 May 2008 08:36:23PM 7 points [-]

1. All a self-improving AI needs is good enough YGBM (you gotta believe me) technology. Eliezer gets this, some of the commenters in this thread don't. The key isn't the ability to solve protein-folding, the key is the ability to manipulate a threshhold amount of us to its ends. 2. We may already functionally be there. You and Chiquita Brands International in a death match. My money is CBI killing you before you kill it. Companies and markets already manipulate our behavior, using reward incentives like the ones in Eliezer's parable for us to engage in behavior that maximizes their persistence at the expense of ours. Whether they're "intelligent", "conscious", or not isn't as relevant to me as the fact that they may have a permanent persistence maximizing advantage over us. My money is probably on corporations and markets substrate jumping and leaving us behind as more likely than us substrate jumping and leaving our cellular and bacteriological medium behind.

Comment author: pnrjulius 09 April 2012 04:51:34AM -1 points [-]

A point well taken. IT professionals have a phrase: "Problem exists between keyboard and chair." Humans are insecure systems; we can be engineered remarkably easily.

Though I do think there is some hope of beating a corporation; you just need a lot of people on your side (a number comparable to the total number of customers, shareholders, and employees of the corporation).