In You Provably Can't Trust Yourself, Eliezer tried to figured out why his audience didn't understand his meta-ethics sequence even after they had followed him through philosophy of language and quantum physics. Meta-ethics is my specialty, and I can't figure out what Eliezer's meta-ethical position is. And at least at this point, professionals like Robin Hanson and Toby Ord couldn't figure it out, either.
Part of the problem is that because Eliezer has gotten little value from professional philosophy, he writes about morality in a highly idiosyncratic way, using terms that would require reading hundreds of posts to understand. I might understand Eliezer's meta-ethics better if he would just cough up his positions on standard meta-ethical debates like cognitivism, motivation, the sources of normativity, moral epistemology, and so on. Nick Beckstead recently told me he thinks Eliezer's meta-ethical views are similar to those of Michael Smith, but I'm not seeing it.
If you think you can help me (and others) understand Eliezer's meta-ethical theory, please leave a comment!
Update: This comment by Richard Chappell made sense of Eliezer's meta-ethics for me.
I agree that by using a single term for the wider class of volitions -- for example, by saying both that I "prefer" autonomy to uniformity and also that I "prefer" male sexual partners to female ones and also that I "prefer" chocolate to vanilla -- I introduce the connotation that the distinctions between these various "preferences" aren't important in the context of discourse.
To call that an unfortunate connotation is question-begging. Sometimes we deliberately adopt language that elides a distinction in a particular context, precisely because we don't believe that distinction ought to be made in that context.
For example, in a context where I believe skin color ought not matter, I may use language that elides the distinction between skin colors. I may do this even if I care about that distinction: for example, if I observe that I do, in fact, care about my doctor's skin color, but I don't endorse caring about it, I might start using language that elides that distinction as a way of changing the degree to which I care about it.
So it seems worth asking whether, in the particular context you're talking about, the connotations introduced by the term "preferences" are in fact unfortunate.
For instance, you class sexual preference among the "chocolate/vanilla" preferences for which the implication that they "shouldn't really matter" is appropriate.
I would likely have agreed with you twenty years ago, when I had just broken up with my girlfriend and hadn't yet started dating my current husband. OTOH, today I would likely "flee screaming" from a ray that made me heterosexual, since that would vastly decrease the value to me of my marriage.
Of course, you may object that this sort of practical consequence isn't what you mean. But there are plenty of people who would "flee screaming" from a sexual-preference-altering ray for what they classify as moral reasons, without reference to practical consequences. And perhaps I'm one of them... after all, it's not clear to me that my desire to preserve my marriage isn't a "moral value."
Indeed, it seems that there simply is no consistent fact of the matter as to whether my sexual preference is a "flee screaming" thing or not... it seems to depend on my situation. 20-year-old single me and 40-year-old married me disagree, and if tomorrow I were single again perhaps I'd once again change my mind.
Now, perhaps that just means that for me, sexual preference is a mere instrumental value, best understood in terms of what other benefits I get from it being one way or another, and is therefore a poor example of the distinction you're getting at, and I should pick a different example.
On the other hand, just because I pick an different preference P such that I can't imagine how a change in environment or payoff matrix might change P, doesn't mean that P actually belongs in a different class from sexual preference. It might be equally true that a similarly pragmatic change would change P, I just can't imagine the change that would do it.
Perhaps, under the right circumstances, I would not wish to flee from an autonomy/uniformity switching ray.
My point is that it's not clear to me that it's a mistake to elide over the distinction between moral values and aesthetic preferences. Maybe calling all of these things "preferences" is instead an excellent way of introducing the fortunate connotation that the degree to which any of them matter is equally arbitrary and situational, however intense the feeling that some preferences are "moral values" or "terminal values" or whatever other privileged term we want to apply to them.
These are two different people, many objections from the fact they disagree one ought to have from the fact that one and some random other contemporary person disagree.