(Replying again here rather than at the foot of a nugatory meta-discussion.)
I suggested C.S. Lewis' "The Abolition of Man" as proposing a candidate for an optimum towards which moral systems have gravitated.
C.S. Lewis was, as Tim Tyler points out, a Christian, but I shall trust that we are all rational enough here to not judge the book from secondary data, when the primary source is so short, clearly written, and online. We need not don the leather cloak and posied beak to avoid contamination from the buboes of this devilish theist oozing Christian memes. It is anyway not written from a Christian viewpoint. To provide a summary would be to make soup of the soup. Those who do not wish to read that, are as capable of not reading this, which is neither written from a Christian viewpoint, nor by a Christian.
I am sufficiently persuaded that the eight heads under which he summarises the Tao can be found in all cultures everywhere: these are things that everyone thinks good. One might accuse him of starting from New Testament morality and recognising only that in his other sources, but if so, the defects are primarily of omission. For example, his Tao contains no word in praise of wisdom: such words can be found in the traditions he draws on, but are not prominent in the general doctrines of Christianity (though not absent either). His Tao is silent on temperance, determination, prudence, and excellence.
Those unfamiliar with talk of virtue can consult this handy aide-memoire and judge for themselves which of them are also to be found in all major moral systems and which are parochial. Those who know many languages might also try writing down all the names of virtues they can think of in each language: what do those lists have in common?
Here's an experiment for everyone to try: think it good to eat babies. Don't merely imagine thinking that: actually think it. I do not expect anyone to succeed, any more than you can look at your own blood and see it as green, or decide to believe that two and two make three.
What is the source of this universal experience?
Lewis says that the Tao exists, it is constant, and it is known to all. People and cultures differ only in how well they have apprehended it. It cannot be demonstrated to anyone, only recognised. He does not speculate in this work on where it comes from, but elsewhere he says that it is the voice of God within us. The less virtuous among us are those who hear that voice more faintly; the evil are those who do not hear it at all, or hear it and hate it. I think there will be few takers for that here.
Some -- well, one, at least -- reverse the arrow, saying that God is the good that we do, which presumably makes Satan the evil that we do.
Others say that there are objective moral facts which we discern by our moral sense, just as we discern objective physical facts by our physical senses; in both cases the relationship requires some effort to attain to the objective truth.
Others say, this is how we are made: we are so constituted as to judge some things virtuous, just as we are so constituted as to judge some things red. They may or may not give evpsych explanations of how this came to be, but whatever the explanation, we are stuck with this sense just as much as we are stuck with our experience of colour or of mathematical truth. We may arrive at moral conclusions by thought and experience, but cannot arbitrarily adopt them. Some claim to have discarded them altogether, but then, some people have managed to put their eyes out or shake their brains to pieces.
Come the Singularity, of course, all this goes by the board. Friendliness is an issue beyond just AGI.
In this video, Julian Savulescu from the Uehiro centre for Practical Ethics argues that human beings are "Unfit for the future" - that radical technological advance, liberal democracy and human nature will combine to make the 21st century the century of global catastropes, perpetrated by terrorists and psychopaths, with tools such as engineered viruses. He goes on to argue that enhanced intelligence and a reduced urge to violence and defection in large commons problems could be achieved using science, and may be a way out for humanity.
Skip to 1:30 to avoid the tedious introduction
Genetically enhance humanity or face extinction - PART 1 from Ethics of the New Biosciences on Vimeo.
Genetically enhance humanity or face extinction - PART 2 from Ethics of the New Biosciences on Vimeo.
Well, I have already said something rather like this. Perhaps this really is a good idea, more important, even, than coding a friendly AI? AI timelines where super-smart AI doesn't get invented until 2060+ would leave enough room for human intelligence enhancement to happen and have an effect. When I collected some SIAI volunteers' opinions on this, most thought that there was a very significant chance that super-smart AI will arrive sooner than that, though.
A large portion of the video consists of pointing out the very strong scientific case that our behavior is a result of the way our brains are structured, and that this means that changes in our behavior are the result of changes in the way our brains are wired.