I think it is insufficiently detailed to identify a unique utility function - it needs to have specific extrapolation and reconciliation procedures filled in, the details of those procedures are important and affect the result, and a bad extrapolation procedure could produce arbitrary results.

That said, programming an AI with any value system that didn't match the template of CEV (plus details) would be a profoundly stupid act. I have seen so many disastrously buggy attempts to define what human values are that I doubt it could be done correctly without the aid of a superintelligence.

So there is no normative rule that Pavitra (you) should care about G101. It just happens, it could also be different and it does not matter. That is what I call (moral) nihilism.

Don't you ever ask why you should care (about anything, incl. yourself caring about things)? (I am not suggesting you becoming suicidal, but on the other hand, there is no normative rule against it, so... hm... I still won't)

Again, it's not that I don't care about anything. I just happen to have a few core axioms, things that I care about for no reason. They don't feel arbitrary to me -- after all, I care about them a great deal! -- but I didn't choose to care about them. I just do.

A large group of crazies agreeing: Ever heard of religion, homeopathy, TCM et cetera?

Sure, and those are the claims I take the time to evaluate and debunk.

If you get into the car, you are a G701, if not, you are a G702.

Please explain the relationship between G701-702 and G698-700.

I struggled to understand your original descriptions, so I rephrased them. Does this capture your categorization?

1. Strong ontological reductionist:
   • a) everything can be reduced to a mathematical object in a mathematical realm
   • b) nothing exists outside the mathematical realm
2. Weak ontological reductionist:
   • a) mental phenomena are entirely physical in nature
   • b) everything does not necessarily reduce to mathematical objects in a mathematical realm
3. Strong scientific reductionist:
   • a) the behavior of the universe is fundamentally deterministic
4. Weak scientific reductionist:
   • a) the behavior of the universe is fundamentally probabilistic
   • b) it is impossible to predict specific outcomes in quantum level systems
   • c) concepts and theories in chemistry and biology are useful high level approximations of the fundamentally probabilistic universe

G101: Pavitra (me) cares about something.

What is the point in caring for G101?

Since I'm Pavitra, it doesn't really matter to me if G101 has a point; I care about it anyway.

What if you can't predict?

Their claims are basically noisy. If a large group of crazies started agreeing with each other, that might require looking into more carefully.

That is not how your brain works (a rough guess).

Not natively, no. That's why it requires advocacy.

So your point is there is no point in caring for anything. Do you call yourself a nihilist?

No, I care about things. It's just that I don't think that G695 (assuming it's defined -- see below) would be particularly humane or good or desirable, any more than (say) Babyeater morality.

Would you call yourself a naive realist?

Certainly not -- hence "eventually". Science requires interpreting data.

Edit: oh, sorry, forgot to address your actual point.

At a certain point, the working model of reality begins to predict what the insane will claim to perceive and how those errors come about.

I would say, the optimistic view is saying "There is probably/hopefully no crash". But don't let us fight over words.

Very well. Let us assume that (warning: numbers just made up) one in every 100,000 car trips results in a crash. The G698 view says "The chances of a crash are low." The G699 view says "The chances of a crash are high." The G700 view says "The chances of a crash are 1/1000000." I advocate the G700 view, and assert that believing G698 or G699 interferes with believing G700.

Does CEV of humankind exists?

I personally don't think the extrapolated volition of humanity coheres, but I have the impression that others disagree with me.

I would be very surprised, however, if the extrapolated volition of all volitional entities cohered and the extrapolated volition of all volitional humans did not.

Absolute morality is the coherent extrapolated volition of all entities with volition.

This sounds like a definition, so let's gensym it and see if it still makes sense.

G695 is the coherent extrapolated volition of all entities with volition.

Why should I care about G695? In particular, why should I prefer it over G696, which is the CEV of all humans with volition alive in 2010, or over G697, which is the CEV of myself?

So the reason why you think there is a reality is because there is a strong consensus and the reason why you think that there is no morality is because there is no strong consensus?

No, that's my reason for breaking symmetry between them, for discarding the assumption that the explanation of the two phenomena should be essentially isomorphic. I then investigate the two unrelated phenomena individually and eventually come to the conclusion that there is one reality between all humans, but a separate morality for each human.

within the boundaries of my ignorance, I can hope for the better or believe in the worse.

There is a very great difference between hoping for the better and believing in the better. Nor are "better" or "worse" the only two options.

Suppose you're getting into a car, and you're wondering whether you will get into a crash. The optimistic view is that you will definitely not crash. The pessimistic view is that you will definitely crash. Neither of these is right.

To be absolute, it has to apply to all mind that has volition.

You're constructing a universal CEV. It's not an already-existing ontologically fundamental entity. It's not a thing that actually exists.

That is why I evaluate arguments based on other things than someone's ulterior motives.

Consciously, sure. I just wanted to warn you against the human credulity bias.

And yet you seem to acknowledge that the output of the CEV function depends on whose volition it is asked to extrapolate. In what sense then is morality absolute, rather than relative to a certain kind of mind?

To be absolute, it has to apply to all mind that has volition.

No Universally Compelling Arguments contains a proof that for every possible morality, there is a mind with volition to which it does not apply. Therefore, there is no absolute morality.

Often, when I stop to think about a decision, I find that my desire changes upon reflection. The latter desire generally seems more intellectually coherent(*), and across multiple instances, the initial desires on various occasions are generally more inconsistent with one another while the after-reflection desires are generally more consistent with one another. From this I infer the existence of a (possibly only vague, partially specified, or partially consistent) common cause to the various instances' after-reflection desires. This common cause appears to roughly resemble a bundle of heuristics that collectively approximate some sort of optimization criteria. I call the bundle of heuristics my "moral intuition" and the criteria they approximate my "morality".

I suspect that other human's minds are broadly similar to mine in this respect, and that their moral intuitions are broadly similar to mine. To the extent they correlate, we might call the set of common trends "human morality" or "humaneness".

(*) An example of intellectual coherence vs. incoherence: Right now, I'd like to go get some ice cream from the freezer. However, on reflection, I remember that there isn't any ice cream in the freezer at the moment, so walking over to the freezer would not satisfy the impulse that motivated the action.