We've got moral intuitions; our initial conditions.

We've got values, which are computed, and morality, our decision making computation.

Our morality updates our values, our values inform our morality, and the particular framework we use, evolves over time.

Do all locally-moral explorations of framework-space converge, even assuming the psychological unity of humans? Our morals influence our psychology; can we ignore the effect of our local morality on the psychology of our future-selves?

Eliezer isn't tempting us with a perfect morality; he's unraveling a meta-ethics, a computation for COMPUTING the evolution of morality, i.e. a framework for iteratively building better moralities.

Why assume that even with this meta-ethics, our morality-evolution converges, rather than diverges (or merely remaining as diverse as it currently is)? Maybe it doesn't matter. We've already been warned against the dichotomy between "morality-as-given" and "morality-as-preference". Morality is not a fixed, immutable structure to which our moral utility-functions will all inevitably converge. But there is a general framework within which we can evaluate moralities, analogous to the framework within which mathematicians explore various formal theories (which seems basically correct). But neither is morality merely a preference, again analogous in my mind to the fact that not all possible mathematical theories are 'interesting'. I think Eliezer needs to fill us in on what makes a morality 'interesting'. Oddly enough, in mathematics at least, there is an intuitive notion of 'interesting' based on the consequences of a formal theory; what theorems does said theory generate?

Certainly, we should be able to exclude certain moralities easily; we've got bedrock 'neath us, right?