Though Friendly AI researchers seem only dimly aware of this, they are actually not the first to argue over which system of ethics is best — and those prior efforts have hardly met with consensus.

[...]

Simply picking certain outcomes — like pain, death, bodily alteration, and violation of personal environment — and asserting them as absolute moral wrongs does nothing to resolve the difficulty of ethical dilemmas in which they are pitted against each other (as, fully understood, they usually are). Friendly AI theorists seem to believe that they have found