Comment on What is Evil about creating House Elves?

Tue, 14 Dec 2010 14:55:59 GMT

you seemed to be claiming, not that being able to usefully interpret natural-language commands entails understanding a great deal about the world, but that it entails resenting oppression. ... I'm interested in your reasons for believing it.

I'm not sure I can do the topic justice without writing a full article, but here's my thinking: to sufficiently locate the "hypothesis" of what the Elf should be doing (given a human command), it requires some idea of human motives. Once it internally represents these human motives well enough to perform like a human servant, and sorts its own preferences the same way, then it has become a human in every way except that it distinguishes between humans and Elves, the former of which should be favored.

But once the Elf has absorbed this human-type generating function, then anything that motivates humans can motivate the Elf -- including sympathizing with servant groups (literally: "believing that it would be good to help the servants gain rank relative to their masters"), and recognizing themselves, or at least other Elves, as being such a servant group.

You can patch these problems, of course, but each patch either makes the Elf more human (and thus wrong to treat as a servant class) or less effective at serving humans. For example, you could introduce a sort of blind spot that makes it ("mechanically") output "that's okay" whenever it observes treatment that it would regard as bad if done to a human. But if this is all that distinguishes Elves from humans, then the Elves start to bear too much cognitive similarity to humans who have undergone psychological abuse.

Discuss