NMJablonski comments on The Amazing Virgin Pregnancy - Less Wrong

22 Post author: Eliezer_Yudkowsky 24 December 2007 02:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (271)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: NMJablonski 01 May 2011 04:40:54PM *  0 points [-]

That one agent's preferences differ greatly from the norm does not automatically make cooperation impossible.

I wasn't arguing that cooperation is impossible. From everything you said there it looks like your understanding of morality is similar to mine:

Agents each judging possible outcomes based upon subjective values and taking actions to try to maximize those values, where the ideal strategy can vary between cooperation, competition, etc.

This makes sense I think when you say:

For example, a society confronting a would-be suicide bomber will (morally and practically) incarcerate him

The members of that society do that because they prefer the outcome in which he does not suicide attack them, to one where he does.

once thwarted from his primary goal, the would-be bomber may find that he now has some common interests with his captors

This phrasing seems exactly right to me. The would-be bomber may elect to cooperate, but only if he feels that his long-term values are best fulfilled in that manor. It is also possible that the bomber will resent his captivity, and if released will try again to attack.

If his utility function assigns (carry out martyrdom operation against the great enemy) an astronomically higher value than his own survival or material comfort, it may be impossible for society to contrive circumstances in which he would agree to long term cooperation.

This sort of morality, where agents negotiate their actions based upon their self-interest and the impact of others actions, until they reach an equilibrium, makes perfect sense to me.