I see morality as fundamentally a way of dealing with conflicts between values/goals, so I cant answer questions posed in terms of "our values", because I don't know whether that means a set of identical values, a set of non-identical but non conflicting values, or a set of conflicting values. One of the implications of that view is that some values/goals are automatically morally irrelevant , since they can be satisfied without potential conflict. Another implication is that my view approximates to "morality is society's rules", but without the dismissive implication..if a society as gone through a process of formulating rules that are effective at reducing conflict, then there is a non-vacuous sense in which that society's morality is its rules. Also AI and alien morality are perfectly feasible, and possibly even necessary.

Joi Ito said several things that are unpleasant but are probably believed by most people, and so I am glad for the reminder.

JOI ITO: This may upset some of my students at MIT, but one of my concerns is that it’s been a predominately male gang of kids, mostly white, who are building the core computer science around AI, and they’re more comfortable talking to computers than to human beings. A lot of them feel that if they could just make that science-fiction, generalized AI, we wouldn’t have to worry about all the messy stuff like politics and society. They think machines will just figure it all out for us.

Yes, you would expect non-white, older, women who are less comfortable talking to computers to be better suited dealing with AI friendliness! Their life experience of structural oppression helps them formally encode morals!

ITO: [Temple Grandin] says that Mozart and Einstein and Tesla would all be considered autistic if they were alive today. [...] Even though you probably wouldn’t want Einstein as your kid, saying “OK, I just want a normal kid” is not gonna lead to maximum societal benefit.

I should probably get a good daily reminder most people would not, in fact, want their kid to be as smart, impactful and successful in life as Einstein, and prefer "normal", not-too-much-above-average kids.

This is an explanation of Yudkowsky's idea from the metaethics sequence. I'm just trying to make it accessible in language and length with lots of concept handles and examples.

Technically, you could believe that people are equally allowed to be enslaved. All people equal + it's wrong to make me a slave = it's wrong to make anyone a slave.

"All men are created equal" emerges from two or more basic principles people are born with. You might say: "Look, you have value, yah? And your loved ones? Would they stop having value if you forgot about them? No? They have value whether or not you know them? How did you conclude they have value? Could that have happened with other people, too? Would you then think they had value? Would they stop having value if you didn't know them? No? Well, you don't know them; do they have value?

You take "people I care about have value" (born with it) and combine it with "be consistent" (also born with), and you get "everyone has value."

That's the idea in principle, anyway. You take some things people are all born with, and they combine to make the moral insights people can figure out and teach each other, just like we do with math.

You're right; I've provided no evidence.

Do you think the idea is sufficiently coherent and non-self-contradictory that the way to find out if it's right or wrong is to look for evidence?

If it was incoherent or contradicted itself, it wouldn't even need evidence to be disproven; we would already know it's wrong. Have I avoided being wrong in that way?

(by the way, understanding slavery might be necessary, but not sufficient to get someone to be against it. They might also need to figure out that people are equal, too. Good point, I might need to add that note into the post).

What?? Weird!

Maybe it was lost when i edited the draft.

I just deleted the link post and made a better one: http://lesswrong.com/r/discussion/lw/o0i/barack_obamas_opinions_on_nearfuture_ai_fixed/

I don't see a link. Was it lost like in my link post on a different subject? I still don't know how to post links correctly.

Without commenting on whether this presentation matches the original metaethics sequence (with which I disagree), this summary argument seems both unsupported and unfalsifiable.

1. No evidence is given for the central claim, that humans can and are converging towards a true morality we would all agree about if only we understood more true facts.
2. We're told that people in the past disagreed with us about some moral questions, but we know more and so we changed our minds and we are right while they were wrong. But no direct evidence is given for us being more right. The only way to judge who's right in a disagreement seems to be "the one who knows more relevant facts is more right" or "the one who more honestly and deeply considered the question". This does not appear to be an objectively measurable criterion (to say the least).
3. The claim that ancients, like Roman soldiers, thought slavery was morally fine because they didn't understand how much slaves suffer is frankly preposterous. Roman soldiers (and poor Roman citizens in general) were often enslaved, and some of them were later freed (or escaped from foreign captivity). Many Romans were freedmen or their descendants - some estimate that by the late Empire, almost all Roman citizens had at least some slave ancestors. And yet somehow these people, who both knew what slavery was like and were often in personal danger of it, did not think it immoral, while white Americans in no danger of enslavement campaigned for abolition.

Nobody is calling “a universal decision theory a moral theory”. According to hedonistic utilitarianism, and indeed all consequentialism, all actions are morally significant.

‘Moral’ means regarding opinions of which actions ought to be performed.