People aren't very good at being utilitarians when there's heavy emotional issues involved even if they are generally good at thinking rationally in other situations. For example, I'm generally a utilitarian, but when I read about this extremely disturbing story I wanted the people responsible to suffer badly for a very long time. And I still do. I don't just want them to die to prevent future harm. I want them to burn. I want them to burn so much that it almost makes me wish there were a vengeful god to torture them. And if I had the choice between simply killing the people involved and making them die slow, agonizing deaths, I'd likely pick the second and them lie to myself and convince myself that that was somehow the utilitarian thing to do.
Humans have a lot of trouble being good utilitarians when the stakes are high.
[Update: and now there's a fifth discussion thread, which you should probably use in preference to this one. Later update: and a sixth -- in the discussion section, which is where these threads are living for now on. Also: tag for HP threads in the main section, and tag for HP threads in the discussion section.]
The third discussion thread is above 500 comments now, just like the others, so it's time for a new one. Predecessors: one, two, three. For anyone who's been on Mars and doesn't know what this is about: it's Eliezer's remarkable Harry Potter fanfic.
Spoiler warning and helpful suggestion (copied from those in the earlier threads):
Spoiler Warning: this thread contains unrot13'd spoilers for Harry Potter and the Methods of Rationality up to the current chapter and for the original Harry Potter series. Please continue to use rot13 for spoilers to other works of fiction, or if you have insider knowledge of future chapters of Harry Potter and the Methods of Rationality.
A suggestion: mention at the top of your comment which chapter you're commenting on, or what chapter you're up to, so that people can understand the context of your comment even after more chapters have been posted. This can also help people avoid reading spoilers for a new chapter before they realize that there is a new chapter.