Tangentially: the first time I read this comment I parsed the first word as "Philosophers." Which rendered the comment puzzling, but not necessarily wrong.
It is not entirely clear to me that Pebblesorters are good standins for humans in this sort of analogy.
But, leaving that aside... applying Wei Dai's argument to Pebblesorters involves asking whether Idealized Pebblesorters use words like "right" and "should" and "good" and "correct" and "proper" with respect to prime-numbered piles, the way Base Pebblesorters do.
I'm not sure what the answer to that question is. It seems to me that they just confuse themselves by doing so, but I feel that way about humans too.
You're certainly right that the computation that encodes their preferences doesn't involve words, but I don't know what that has to do with anything. The computation that encodes our preferences doesn't involve words either... and so?
The further along this track I go, the less meaningful the question seems. I guess I'm Just Not Getting It.
I think I've found a better argument that Eliezer's meta-ethics is wrong. The advantage of this argument is that it doesn't depend on the specifics of Eliezer's notions of extrapolation or coherence.
Eliezer says that when he uses words like "moral", "right", and "should", he's referring to properties of a specific computation. That computation is essentially an idealized version of himself (e.g., with additional resources and safeguards). We can ask: does Idealized Eliezer (IE) make use of words like "moral", "right", and "should"? If so, what does IE mean by them? Does he mean the same things as Base Eliezer (BE)? None of the possible answers are satisfactory, which implies that Eliezer is probably wrong about what he means by those words.
1. IE does not make use of those words. But this is intuitively implausible.
2. IE makes use of those words and means the same things as BE. But this introduces a vicious circle. If IE tries to determine whether "Eliezer should save person X" is true, he will notice that it's true if he thinks it's true, leading to Löb-style problems.
3. IE's meanings for those words are different from BE's. But knowing that, BE ought to conclude that his meta-ethics is wrong and morality doesn't mean what he thinks it means.