I'm not getting the essential difference between (3) and (4). It seems like (4) is just a special case of (3), in that (4)'s "process of reflection [that] is convergent at the individual level" could just be (3)'s "how to translate non-preferences ... into preferences".
Is the difference that, if (3) held, then there could be only one correct "process of reflection" of the kind described in (4), so that this one process would be the only correct path for all pre-rational intelligent agents to take to become rational?
Is the difference that, if (3) held, then there could be only one correct "process of reflection" of the kind described in (4), so that this one process would be the only correct path for all pre-rational intelligent agents to take to become rational?
Yes, or at least there are right and wrong ways to reflect on what one wants, so that even if someone were to reach reflective equilibrium via a convergent process, it would make sense to say that they did it wrong and ended up with wrong values (or "wrong values for them").
In this post, I list six metaethical possibilities that I think are plausible, along with some arguments or plausible stories about how/why they might be true, where that's not obvious. A lot of people seem fairly certain in their metaethical views, but I'm not and I want to convey my uncertainty as well as some of the reasons for it.
(Note that for the purposes of this post, I'm concentrating on morality in the axiological sense (what one should value) rather than in the sense of cooperation and compromise. So alternative 1, for example, is not intended to include the possibility that most intelligent beings end up merging their preferences through some kind of grand acausal bargain.)
It may be useful to classify these possibilities using labels from academic philosophy. Here's my attempt: 1. realist + internalist 2. realist + externalist 3. relativist 4. subjectivist 5. moral anti-realist 6. normative anti-realist. (A lot of debates in metaethics concern the meaning of ordinary moral language, for example whether they refer to facts or merely express attitudes. I mostly ignore such debates in the above list, because it's not clear what implications they have for the questions that I care about.)
One question LWers may have is, where does Eliezer's metathics fall into this schema? Eliezer says that there are moral facts about what values every intelligence in the multiverse should have, but only humans are likely to discover these facts and be motivated by them. To me, Eliezer's use of language is counterintuitive, and since it seems plausible that there are facts about what everyone should value (or how each person should translate their non-preferences into preferences) that most intelligent beings can discover and be at least somewhat motivated by, I'm reserving the phrase "moral facts" for these. In my language, I think 3 or maybe 4 is probably closest to Eliezer's position.