Zubon comments on No Universally Compelling Arguments - Less Wrong

33 Post author: Eliezer_Yudkowsky 26 June 2008 08:29AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (46)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Zubon 26 June 2008 02:55:30PM 5 points [-]

Presumably, Eliezer's simulated planet of Einsteins would be able to infer everything about the tentacle-creatures' morality simply based on the creatures' biology and evolutionary past. So I think this hypothetical super-AI could in fact figure out what morality humans subscribe to.

Is anyone else terrified by the notion of a super-AI that looked at human history and figured out what morality humans actually subscribed to, rather than claimed to, and then instantiated it? I can only hope that the AI gives enough weight to the people who lived the sorts of quiet lives that don't make the history books. Looking to our evolutionary past is also a scary notion.