Ok, let's take 1 more seriously. In order for Eliezer's meta-ethics to qualify as meta-ethics, he has to at least roughly specify what IE is. But how do you specify an idealized version of yourself that reasons about morality without using words like "moral", "right" and "should"? If Eliezer takes Base Eliezer and just deletes the parts of his mind that are related to these words, he's almost certainly not going to like the results. What else could he do?
But how do you specify an idealized version of yourself that reasons about morality without using words like "moral", "right" and "should"?
You don't use those words, you refer to your brain as a whole, which happens to already contain those things, and specify extrapolation operations like time passing that it might go through. (Note that no one has nailed down what exactly the ideal extrapolation procedure would be, although there's some intuition about what is and isn't allowed. There is an implied claim there that differ...
I think I've found a better argument that Eliezer's meta-ethics is wrong. The advantage of this argument is that it doesn't depend on the specifics of Eliezer's notions of extrapolation or coherence.
Eliezer says that when he uses words like "moral", "right", and "should", he's referring to properties of a specific computation. That computation is essentially an idealized version of himself (e.g., with additional resources and safeguards). We can ask: does Idealized Eliezer (IE) make use of words like "moral", "right", and "should"? If so, what does IE mean by them? Does he mean the same things as Base Eliezer (BE)? None of the possible answers are satisfactory, which implies that Eliezer is probably wrong about what he means by those words.
1. IE does not make use of those words. But this is intuitively implausible.
2. IE makes use of those words and means the same things as BE. But this introduces a vicious circle. If IE tries to determine whether "Eliezer should save person X" is true, he will notice that it's true if he thinks it's true, leading to Löb-style problems.
3. IE's meanings for those words are different from BE's. But knowing that, BE ought to conclude that his meta-ethics is wrong and morality doesn't mean what he thinks it means.