Boiling it down to essentials, it looks to me like the key move is this:

• If we can prove X, then we can prove Y.
• Therefore, if X is true, then we can prove Y.

But this doesn't follow - X could be true but not provable.

Is that right? It's ages since I did logic, and never to a deep level, so excuse me if this is way off.

Eliezer, I think I kind-of understand by now why you don't call yourself a relativist. Would you say that it's the "psychological unity of mankind" that distinguishes you from relativists?

A relativist would stress that humans in different cultures all have different - though perhaps related - ideas about "good" and "right" and so on. I believe your position is that the bulk of human minds are similar enough that they would arrive at the same conclusions given enough time and access to enough facts; and therefore, that it's an objective matter of fact what the human concepts of "right" and "good" actually mean.

And since we are human, there's no problem in us continuing to use those words.

Am I understanding correctly?

It seems like your position would become more akin to relativism if the "psychological unity" turned out to be dubious, or if our galaxy turned out to be swarming with aliens, and people were forced to deal with genuinely different minds. In those cases, would there still be anything to separate you from actual relativists?

(In either case, it would still be an objective matter of fact what any given mind would call "good" if given enough time - but that would be a much less profound fact than it is for a species all alone and in a state of psychological unity.)

It's a datum (which any adequate metaethical theory must account for) that there can be substantive moral disagreement. When Bob says "Abortion is wrong", and Sally says, "No it isn't", they are disagreeing with each other.

I wonder though: is this any more mysterious than a case where two children are arguing over whether strawberry or chocolate ice cream is better?

In that case, we would happily say that the disagreement comes from their false belief that it's a deep fact about the universe which ice cream is better. If Eliezer is right (I'm still agnostic about this), wouldn't moral disagreements be explained in an analogous way?

This post hits me far more strongly than the previous ones on this subject.

I think your main point is that it's positively dangerous to believe in an objective account of morality, if you're trying to build an AI. Because you will then falsely believe that a sufficiently intelligent AI will be able to determine the correct morality - so you don't have to worry about programming it to be friendly (or Friendly).

I'm sure you've mentioned this before, but this is more forceful, at least to me. Thanks.

Personally, even though I've mentioned that I thought there might be an objective basis for morality, I've never believed that every mind (or even a large fraction of minds) would be able to find it. So I'm in total agreement that we shouldn't just assume a superintelligent AI would do good things.

In other words, this post drives home to me that, pragmatically, the view of morality you propose is the best one to have, from the point of view of building an AI.

And I may not know what this question is, actually; I may not be able to print out my current guess nor my surrounding framework; but I know, as all non-moral-relativists instinctively know, that the question surely is not just "How can I do whatever I want?"

I'm not sure you've done enough to get away from being a "moral relativist", which is not the same as being an egoist who only cares about his own desires. "Moral relativism" just means this (Wikipedia):

In philosophy, moral relativism is the position that moral or ethical propositions do not reflect objective and/or universal moral truths [...] Moral relativists hold that no universal standard exists by which to assess an ethical proposition's truth.

Unless I've radically misunderstood, I think that's close to your position. Admittedly, it's an objective matter of fact whether some action is good according to the "blob of a computation" (i.e. set of ethical concerns) that any specific person cares about. But there's no objective way to determine that one "blob" is any more correct than another - except by the standards of those blobs themselves.

(By the way, I hope this isn't perceived as particular hostility on my part: I think some very ethical and upstanding people have been relativists. It's also not an argument that your position is wrong.)

Myself: I can't help but wonder about anthropic effects here. It might be the case that nuclear-armed species annihilate themselves with high probability (say 50% per decade), but of course, all surviving observers live on planets where it hasn't happened through sheer chance.

Just to expand on this (someone please stop me if this sort of speculative post is irritating):

Imagine there are a hundred Earths (maybe because of MWI, or because the universe is infinite, or whatever). Lets say there's a 90% chance of nuclear war before 2008, and such a war would reduce the 2008 population by 90%. In that case, you still end up with 53% of observers in 2008 living on an Earth where nuclear war didn't occur.

This implies that we might be overconfident, and assign too low a probability to nuclear war, just because we've survived as long as we have.

But: The argument seems to implicitly assume that I am a random observer in 2008. I'm not sure this is legitimate. Anthropic reasoning is irritatingly tricky.

Time has passed, and we still haven't blown up our world, despite a close call or two.

I can't help but wonder about anthropic effects here. It might be the case that nuclear-armed species annihilate themselves with high probability (say 50% per decade), but of course, all surviving observers live on planets where it hasn't happened through sheer chance.

(Though on the other hand, if an all-out nuclear war is survivable for a species like ours, then this line of thought wouldn't work.)

Poke, can you expand a little on what you're driving at?

Also, Steven, how on Earth is that statement true under MWI? :)

We do not know very well how the human mind does anything at all. But that the the human mind comes to have preferences that it did not have initially, cannot be doubted.

I believe Eliezer is trying to create "fully recursive self-modifying agents that retain stable preferences while rewriting their source code". Like Sebastian says, getting the "stable preferences" bit right is presumably necessary for Friendly AI, as Eliezer sees it.

(This clause "as Eliezer sees it" isn't meant to indicate dissent, but merely my total incompetence to judge whether this condition is strictly necessary for friendly AI.)

I am assuming [the AI] acts, and therefore makes choices, and therefore has preferences, and therefore can have preferences which conflict with the preferences of other minds (including human minds).

An AI can indeed have preferences that conflict with human preferences, but if it doesn't start out with such preferences, it's unclear how it comes to have them later.

On the other hand, if it starts out with dubious preferences, we're in trouble from the outset.