...is a powerful argument I had never considered.
How would you say this relates to the ethics of creating an FAI? In some ways house elves were created for a similar reason that we would create an FAI. Would it be something about 'consciousness' that separates the two constructions ethically? If so, I wonder whether creating a 'helper' agent that in some sense is conscious and 'enjoys' what it does is better or worse than creating a raw optimised agent that we wouldn't consider conscious.
It occurs to me that what a house elf considers fun is not all that much different from the perspective of all of value-space from what we might consider fun.
ETA: There is now a third thread, so send new comments there.
Since the first thread has exceeded 500 comments, it seems time for a new one, with Eliezer's just-posted Chapter 33 & 34 to kick things off.
From previous post: