Yeah, a lot of the Metaethics Sequence seems to be trying to get to this point.
For my part, it seems easier to just stop using words like "good" if we believe they are likely to be misunderstood, rather than devoting a lot of energy to convincing everyone that they should mean something different by the word (or that the word really means something different from what they think it means, or whatever).
I'm content to say that we value what we currently value, because we currently value it, and asking whether that's good or not is asking an empty question.
Of course, I do understand the rhetorical value of getting to claim that our AI does good, rather than "merely" claiming that it implements what we currently value.
.
I'm content to say that we value what we currently value, because we currently value it, and asking whether that's good or not is asking an empty question.
I am content to say the question is not empty, and if you assumptions lead you to suppose it is, then your assumptions need to be questioned.
There seems to be a widespread impression that the metaethics sequence was not very successful as an explanation of Eliezer Yudkowsky's views. It even says so on the wiki. And frankly, I'm puzzled by this... hence the "apparently" in this post's title. When I read the metaethics sequence, it seemed to make perfect sense to me. I can think of a couple things that may have made me different from the average OB/LW reader in this regard: