As I read should here unqualified it means that natural selection favours certain kinds of beliefs ie those that help prosperity. Althought for some people it also means spesifications on in which prosperity to shoot for. I tend to be very suspicious of claiming one direction of radiation being better than another (before / unrelated to the genocide mechanism).
Although my concept deconstruction might have erased any "should" and might be a bit unstandard. It's not anymore about "good" and "evil" but what is possible and what's impossible. Some actions are highly dangerous and resource depleting being very nearly impossible and need to be offset by a lot of enabling features. Thus it's about which forms of life are maintainable, under which conditions and which have a half-life. In this way whether helium happens more than uranium is the same kind of question whether a code of conduct results in a prosperous society (say two forms of goverment) but just orders of magnitude more harder to answer. But does it mean that uranium is more evil than helium as elements? Humans know how to use uranium as part of global security so as natural resources it clearly can produce a human good.
In this post, I list six metaethical possibilities that I think are plausible, along with some arguments or plausible stories about how/why they might be true, where that's not obvious. A lot of people seem fairly certain in their metaethical views, but I'm not and I want to convey my uncertainty as well as some of the reasons for it.
(Note that for the purposes of this post, I'm concentrating on morality in the axiological sense (what one should value) rather than in the sense of cooperation and compromise. So alternative 1, for example, is not intended to include the possibility that most intelligent beings end up merging their preferences through some kind of grand acausal bargain.)
It may be useful to classify these possibilities using labels from academic philosophy. Here's my attempt: 1. realist + internalist 2. realist + externalist 3. relativist 4. subjectivist 5. moral anti-realist 6. normative anti-realist. (A lot of debates in metaethics concern the meaning of ordinary moral language, for example whether they refer to facts or merely express attitudes. I mostly ignore such debates in the above list, because it's not clear what implications they have for the questions that I care about.)
One question LWers may have is, where does Eliezer's metathics fall into this schema? Eliezer says that there are moral facts about what values every intelligence in the multiverse should have, but only humans are likely to discover these facts and be motivated by them. To me, Eliezer's use of language is counterintuitive, and since it seems plausible that there are facts about what everyone should value (or how each person should translate their non-preferences into preferences) that most intelligent beings can discover and be at least somewhat motivated by, I'm reserving the phrase "moral facts" for these. In my language, I think 3 or maybe 4 is probably closest to Eliezer's position.