First of all, when I (Base Dorikka, BD) am using the phrases "X is moral," "X is right," and "I should X" in this comment, I am hypothesizing that a version of myself with a comparatively large amount of computing resources and information (Idealized Dorikka, ID) would come to the conclusion that action X would optimize ID's utility function in relation to the set of all actions that BD could take in the situation.
Note that I am not stating with that what I say is "moral" would for certain optimize ID's utility function in relation to the set of all actions that BD could take in the situation -- my saying that "X is moral" represents me making a probability estimate that I would do so. It's key to my understanding here that I don't actually have any authority to declare something the moral thing, I am just estimating that the probability that X is moral is high.
This is my understanding of the use of such phrases -- if you disagree with my conclusions, check if we disagree on how Eliezer uses the words.
Now, BD does not, of course, have access to ID's computing resources, and so I can only imagine a version of ID a certain level above my own. However, ID would have enough computing resources to imagine an ID+ that was a certain level above him. If ID+ ever came to exist, he would be able to imagine an ID++, and so on.
I think that Eliezer's metaethics would select option #2, and point out that ID's reference for what he should do is ID+, not himself.
Edit: I, however, do have the objection that it isn't meaningful for BD to make an estimate of what ID's decision would be because he doesn't have ID's computing power. BD could not estimate ID's utility function with any more information that BD has access to, so BD's estimate of ID's utility function is no better than BD's own utility function.
I think I've found a better argument that Eliezer's meta-ethics is wrong. The advantage of this argument is that it doesn't depend on the specifics of Eliezer's notions of extrapolation or coherence.
Eliezer says that when he uses words like "moral", "right", and "should", he's referring to properties of a specific computation. That computation is essentially an idealized version of himself (e.g., with additional resources and safeguards). We can ask: does Idealized Eliezer (IE) make use of words like "moral", "right", and "should"? If so, what does IE mean by them? Does he mean the same things as Base Eliezer (BE)? None of the possible answers are satisfactory, which implies that Eliezer is probably wrong about what he means by those words.
1. IE does not make use of those words. But this is intuitively implausible.
2. IE makes use of those words and means the same things as BE. But this introduces a vicious circle. If IE tries to determine whether "Eliezer should save person X" is true, he will notice that it's true if he thinks it's true, leading to Löb-style problems.
3. IE's meanings for those words are different from BE's. But knowing that, BE ought to conclude that his meta-ethics is wrong and morality doesn't mean what he thinks it means.