...is a powerful argument I had never considered.

How would you say this relates to the ethics of creating an FAI? In some ways house elves were created for a similar reason that we would create an FAI. Would it be something about 'consciousness' that separates the two constructions ethically? If so, I wonder whether creating a 'helper' agent that in some sense is conscious and 'enjoys' what it does is better or worse than creating a raw optimised agent that we wouldn't consider conscious.

It occurs to me that what a house elf considers fun is not all that much different from the perspective of all of value-space from what we might consider fun.