Well, right, but I'm left with the same question. I mean, yes, I agree that "once it internally represents these human motives well enough to perform like a human servant, and sorts its own preferences the same way," then everything else you say follows, at least to a rough approximation.
But why need it sort its own preferences the same way humans do?
What seems to underlie this argument is an idea that no cognitive system can understand a human's values well enough to predict its preferences without sharing those values... that I can't understand what you want well enough to serve you unless I want the same things.
If that's true, it's news to me, so I'm interested in the arguments for it.
For example, it certainly seems possible to model other things in the world without myself becoming those things: I can develop a working model of what pleases and upsets my dog, and what she likely wants me to do when she behaves in certain ways, without myself being pleased or upset or wanting those things. Do you claim that's an illusion?
But why need it sort its own preferences the same way humans do?
That is what I (thought I) was explaining in the following paragraphs. Once it a) knows what humans want, and b) desires acting in a way that matches that preference ranking, it must carve out a portion of the world's ontology that excludes itself from being a recipient of that service.
It's not that the Elf would necessarily want to be served like it serves others (although that is a failure mode too); it's that the Elf would resemble a human well enough at that point that we would have to...
Edit: This is old material. It may be out of date.
I'm talking about the fictional race of House Elves from the Harry Potter universe first written about by J. K. Rowling and then uplifted in a grand act of fan-fiction by Elizer Yudkowsky. Unless severely mistreated they enjoy servitude to their masters (or more accurately the current residents of the homes they are binded to), this is also enforced by magical means since they must follow the letter if not the spirit of their master's direct order.
Overall treating House Elves the way they would like to be treated appears more or less sensible and don't feel like debating this if people don't disagree. Changing agents without their consent or knowledge seems obviously wrong, so turning someone into a servant creatures seem intuitively wrong. I can also understand that many people would mind their descendants being modified in such a fashion, perhaps their dis-utility is enough to offset the utility of their modified descendants. However how true is this of distant descendants that only share passing resemblance? I think a helpful reminder of scale might be our own self domestication.
Assuming one created elf like creatures ex nihilo, not as slightly modified versions of a existing species why would one not want to bring a mind into existence that would value its own existence and benefits you, as long as the act of creation or their existence in itself does not represents huge enough dis-utility? This seems somewhat related to the argument Robin Hanson once made that any creatures that can pay for their own existance and would value their own existance should be created.
I didn't mention this in the many HP fan fiction threads because I want a more general debate on the treatment and creation of such a class of agents.
Edit: Clearly if the species or class contains exceptions there should be ways for them to pursue their differing values.