Eneasz comments on The Bedrock of Fairness

0Eliezer_Yudkowsky03 July 2008 06:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (58)

You are viewing a single comment's thread.

Eneasz03 July 2008 06:29:29AM0 points [-]

I'm about to make a naked assertion with nothing to back it up, just to put it out there.

The purpose of morality is to prevent such an arguement from even ever occurring. If the morale engine of society is working correctly, then all it's members will have a desire for everyone to get an equally sized portion of the pie (in this example). If there is a Zaire who believes he should get 1/2 of the pie, then there was a malfunction when morality was being programmed into him. This malfunction will lead to conflict.

View it like you would view programming a friendly AI. The purpose is to program the AI with desires that will motivate it to help humanity, and to have a strong aversion to destroying humanity. If this goal is not reached, there was a failure by the programmers. I think it's been said on this blog that if you create an AI without having made it friendly you've already lost and the game is over. It's not quite as drastic if you fail with humans, but the principle is the same. If a friendly Human Intelligence is not programmed with the desires that will help to keep humanity thriving then there was a failure by it's programmers(parents/society/teachers/whoever).

Why is it that human morality is this confusing and mysterious realm that no one seems to be able to fathom when AI morality is straight-forward? Is it just that humans can easily see the goal of one (an AI that desires to help rather than hurt humanity) and for some reason can't see the goal of another (a human that desires to help rather than hurt humanity)?