Come to think of it, we may well get to see this exchange. But since we won't see it for a while, I'll try to channel it:
"Harry, taking people's interests into account--being a good guy--requires thinking of them as people. You care a lot about a human, less about a dog, and not at all about a paperclip, right?"
"Right."
"So when you don't treat a human like a human--"
"Hold on. You're equivocating. To 'treat a human like' their desires are as important to my utility function as my own is an absolute good. To 'treat a human like' convention dictates a human should be treated is a contingent good--it only makes sense when that helps them achieve their desires."
"No, they're not the same thing, Harry. But they're closely linked in your head. You have a cluster of concepts, instincts, and behaviors to do with humans, and each bit reinforces each other bit. You can plainly see how it works: if you spend a year pretending that a toy is a person, you'll become incredibly reluctant to take it apart for spare parts. Conversely, if you start acting like people are your toys..."
"Now you're dehumanizing me a bit, Hermione. If I go into an interaction with Padma planning to help her, I'm going to end up doing my best to help her. Because I'm a sentient being who is aware of his own intentions, not a finite state machine that can get accidentally stuck in the mode for dealing with paperclips."
"Well, Harry, I guess you have more faith in yourself than I do. I think you want your utility function to be different from what it is. I think that, like a lot of people, you're more selfish than you want to be."
"That's incoherent."
"Exactly. You're not going to behave in a logically coherent way. It's okay to aspire to do so, I guess, but please realize that right now, you have to be sure not to--"
"Accidentally train myself to be a bad dog rather than a good dog?"
"Not to drift into Evil while trying to be Good. That's the human condition."
"To 'treat a human like' convention dictates a human should be treated is a contingent good--it only makes sense when that helps them achieve their desires"
The obvious thing this seems to miss is that most people do desire to be treated like people -- at the very least as equals, and with dignity and respect.
So treating them otherwise is by itself of negative value -- not just contingent to other consequences.
Update: Discussion has moved on to a new thread.
After 61 chapters of Harry Potter and the Methods of Rationality and 5 discussion threads with over 500 comments each, HPMOR discussion has graduated from the main page and moved into the Less Wrong discussion section (which seems like a more appropriate location). You can post all of your insights, speculation, and, well, discussion about Eliezer Yudkowsky's Harry Potter fanfic here.
Previous threads are available under the harry_potter tag on the main page (or: one, two, three, four, five); this and future threads will be found under the discussion section tag (since there is a separate tag system for the discussion section). See also the author page for (almost) all things HPMOR, and AdeleneDawner's Author's Notes archive for one thing that the author page is missing.
As a reminder, it's useful to indicate at the start of your comment which chapter you are commenting on. Time passes but your comment stays the same.
Spoiler Warning: this thread is full of spoilers. With few exceptions, spoilers for MOR and canon are fair game to post, without warning or rot13. More specifically: