Yes, that makes sense - even if mine is a better description of usage, from the standpoint of someone categorizing beliefs, I imagine yours would be the better metric.

ETA: I'm not sure the caveat is required for "obviously false", for two reasons.

1. Any substantive thesis (a category which includes most theses that are rejected as obviously false) requires less evidence to be roundly disconfirmed than it does to be confirmed.

2. As Yvain demonstrated in Talking Snakes, well-confirmed theories can be "obviously false", by either of our definitions.

It's true that it usually takes less effort to disabuse someone of an obviously-true falsity than to convince them of an obviously-false truth, but I don't think you need a special theory to support that pattern.

If rerunning the clock produces radically different moralities each time, the relativists would be considered to be correct.

If rerunning the clock produces highly similar moralities, then the moral objectivists will be able to declare victory.

Why should we care about this mere physical fact of which you speak? What has this mere "is" to do with whether "should" is "objective", whatever that last word means (and why should we care about that?)

"Universal values" presumably refers to values the universe will converge on, once living systems have engulfed most of it.

If rerunning the clock produces radically different moralities each time, the relativists would be considered to be correct.

If rerunning the clock produces highly similar moralities, then the moral objectivists will be able to declare victory.

Yeah, but Stefan's post was about AI, not about minds that evolved in our universe.

Also, there is a difference between moral universalism and moral objectivism. What your last sentence describes is universalism, while Stefan is talking about objectivism:

"My claim is that compassion is a universal rational moral value. Meaning any sufficiently rational mind will recognize it as such."

The idea that physics makes no mention of morality seems totally and utterly irrelevant to me. Physics makes no mention of convection, diffusion-limited aggregation, or fractal drainage patterns either - yet those things are all universal.

Agreed.

"Universal values" is usually understood by way of an analogy to a universal law of nature. If there are universal values they are universal in the same way f=ma is universal. Importantly this does not mean that everyone at all times will have these values, only that the question of whether or not a person holds the right values can be answered by comparing their values to the "universal values".

There is a separate question about what beliefs about morality people (or more generally, agents) actually hold and there is another question about what values they will hold if when their beliefs converge when they engulf the universe. The question of whether or not there are universal values does not traditionally bear on what beliefs people actually hold and the necessity of their holding them. It could be the case that there are universal values and that, by physical necessity, no one ever holds them. Similarly, there could be universal values that are held in some possible worlds and not others. This is all the result of the simply observation that ought cannot be derived from is. In the above comment you conflate about a half dozen distinct theses.

The idea that physics makes no mention of morality seems totally and utterly irrelevant to me. Physics makes no mention of convection, diffusion-limited aggregation, or fractal drainage patterns either - yet those things are all universal.

But all those things are pure descriptions. Only moral facts have prescriptive properties and while it is clear how convection supervenes on quarks it isn't clear how anything that supervenes on quarks could also tell me what to do. At the very least if quarks can tell you what to do it would be weird and spooky. If you hold that morality is only the set of facts that describe people's moral opinions and emotions (as you seem to) than you are a kind of moral anti-realist, likely a subjectivist or non-cognitivist.

If that is your stated position then in what way can you claim to create FAI with this whimsical set of goals?

Were it within my power to do so I would create a machine that was really, really good at doing things I like. It is that simple. This machine is (by definition) 'Friendly' to me.

you will end up being a deluded self serving optimizer.

I don't know where the 'deluded' bit comes from but yes, I would end up being a self serving optimizer. Fortunately for everyone else my utility function places quite a lot of value on the whims of other people. My self serving interests are beneficial to others too because I am actually quite a compassionate and altruistic guy.

PS: Instead of using quotation marks you can put a '>' at the start of a quoted line. This convention makes quotations far easier to follow. And looks prettier.

There is no such thing as an "unobjectionable set of values".

Imagine the values of an agent that wants all the atoms in the universe for its own ends. It will object to any other agent's values - since it objects to the very existence of other agents - since those agents use up its precious atoms - and put them into "wrong" configurations.

Whatever values you have, they seem bound to piss off somebody.