Many people think you can solve the Friendly AI problem just by writing certain failsafe rules into the superintelligent machine's programming, like Asimov's Three Laws of Robotics. I thought the rebuttal to this was in "Basic AI Drives" or one of Yudkowsky's major articles, but after skimming them, I haven't found it. Where are the arguments concerning this suggestion?
I am aware of ignoring threats, using uncompromisable principles to get an advantage in negotiations, breaking your receiver to decide on a meeting point, breaking your steering wheel to win at Chicken, etc. I am also aware of the theorem that says even if there is a mutually beneficial trade, there are cases where selfish rational agents refuse to trade, and that the theorem does not go away when the currency they use is thousands of lives. I still claim that the type of war I'm talking about doesn't stem from such calculations; that people on side A are willing to trade a death on side A for a death on side B, as evidenced by their decisions, knowing that side B is running the same algorithm.
A non-war exemple is blood feuds; you know that killing a member of family B who killed a member of family A will only lead to perpetuating the feud, but you're honor-bound to do it. Now, the concept of honor did originate from needing to signal a commitment to ignore status exortion, and (in the absence of relatively new systems like courts of law) unilaterally backing down would hurt you a lot - but honor acquired a value of its own, independently from these goals. (If you doubt it, when France tried to ban duels and encourage trials, it used a court composed of war heroes who'd testified the plaintiff wasn't dishonourable for refusing to duel.)
Plausible, but not true of the psychology of this particular case.
Well obviously they aren't foe-deaths-maximizers. It's just that they're willing to trade off a lot of whatever-they-went-to-war-for-at-first in order to annoy the enemy.
The person who said that was talking about a war where it's quite unrealistic to think any side would go away (as with all wars over inhabited territory). Genociding the other side would be outright easier.
Agree it isn't. I don't even think anyone starts a war with that in mind - war is typically a game of Chicken. I'm pointing out a failure that leads from "I'm going to instill my supporters with an irrational burning hatred of the enemy, so that I can't back down, so that they have to" to "I have an irrational burning hatred of the enemy! I'll never let them back down, that'd let them off too easily!".
Care to guess which war in particular I was thinking of? (By PM if it's too political.) I think it applies to any entrenched conflict where the identify as enemies of the and have done so for several generations, but I do have a prototype. Hints:
I'll go along, but don't forget my original point was that this psychology does not universally characterize war.
Good point, you are right about that.
... (read more)