even if someone were to reach reflective equilibrium via a convergent process, it would make sense to say that they did it wrong and ended up with wrong values (or "wrong values for them").
Thanks. I can sort-of imagine that some, but not all, ways of reaching reflective equilibrium could be "wrong", even if the values held in that equilibrium state could not be said to be "wrong".
But, under hypotheses (3), that's the most we could say, right? How could we go on to say that the agent ended up with "wrong values" if, under hypothesis (3), there is no fact of the matter about which values are "wrong"?
Or maybe a scenario (2.5) could be added intermediate between your (2) and your (3). Under this scenario, as in (3), there are no facts about what everyone should value. Nonetheless, for each individual agent, there is a fact about what that agent should value. However, as in (2), the typical agent will not converge exactly on its "correct" values. Instead, the typical agent will converge on its values "along with idiosyncratic values".
In this post, I list six metaethical possibilities that I think are plausible, along with some arguments or plausible stories about how/why they might be true, where that's not obvious. A lot of people seem fairly certain in their metaethical views, but I'm not and I want to convey my uncertainty as well as some of the reasons for it.
(Note that for the purposes of this post, I'm concentrating on morality in the axiological sense (what one should value) rather than in the sense of cooperation and compromise. So alternative 1, for example, is not intended to include the possibility that most intelligent beings end up merging their preferences through some kind of grand acausal bargain.)
It may be useful to classify these possibilities using labels from academic philosophy. Here's my attempt: 1. realist + internalist 2. realist + externalist 3. relativist 4. subjectivist 5. moral anti-realist 6. normative anti-realist. (A lot of debates in metaethics concern the meaning of ordinary moral language, for example whether they refer to facts or merely express attitudes. I mostly ignore such debates in the above list, because it's not clear what implications they have for the questions that I care about.)
One question LWers may have is, where does Eliezer's metathics fall into this schema? Eliezer says that there are moral facts about what values every intelligence in the multiverse should have, but only humans are likely to discover these facts and be motivated by them. To me, Eliezer's use of language is counterintuitive, and since it seems plausible that there are facts about what everyone should value (or how each person should translate their non-preferences into preferences) that most intelligent beings can discover and be at least somewhat motivated by, I'm reserving the phrase "moral facts" for these. In my language, I think 3 or maybe 4 is probably closest to Eliezer's position.