Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Patrick_(orthonormal) comments on Emotional Involvement - Less Wrong

12 Post author: Eliezer_Yudkowsky 06 January 2009 10:23PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (53)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Patrick_(orthonormal) 07 January 2009 08:02:48PM 0 points [-]

A rogue paperclipper in a mostly Friendly world can probably only be stopped by racial prejudice--to a rational creature, it's always easier to feed him your neighbor than it is to fight him.

A couple of problems with this statement, as I see it:

1. The word "only". Forget five minutes— think for five seconds about Third Alternatives. At the very least, wouldn't an emotion for human-favoritism serve the goal better than an emotion for race-favoritism? Then everyone could cooperate more fully, not just each race by itself.

You could be using "racial prejudice" to mean "species prejudice" or something even wider, but that's not what the question's about. Your argument gives no reason for maintaining the current brain architecture, which creates these divisions of allegiance within the normal human race.

2. Rational agents are doomed to fail because they won't cooperate enough? I stand with Eliezer: rational agents should WIN. If the inevitable result of noncooperation is eventual destruction, genuinely rational agents WILL find ways to cooperate; the Prisoner's Dilemma doesn't operate within every conceivable cooperative enterprise.