The thing is that people aren't perfect rationalists, and part of being a good rationalist is acknowledging your own flaws and limitations.
If you accept to kill, you'll kill, even in situations where killing wasn't necessary, because you'll stop searching the hypothesis space when you find a solution that involves killing. Or because you'll estimate that killing one will save two, but your estimation was flawed - you killed one, and yet the two still die. And it's also something you should know about the way humans work, that once you did something once, it's easier to do it again - and the killing curse seems to model that quite well.
Harry putting himself a "I'll not kill" rule is him forcing himself to find solutions that don't require killing. Especially when you see how his "dark side" work, finding solutions to "impossible" problems when really pressured to do it, it doesn't seem irrational from him to test the hypothesis that he, with his rationalist training, and his "dark side" creativity, can find solutions that don't involve killing. And that only if that hypothesis is falsified, he'll resort to killing.
I think Harry's mistake is that he has left himself no setting between, "no killing" and "all bets are off".
Edit: New thread posted here.
This is a new thread to discuss Eliezer Yudkowsky’s Harry Potter and the Methods of Rationality and anything related to it. This thread is intended for discussing chapter 86. The previous thread has long passed 500 comments.
There is now a site dedicated to the story at hpmor.com, which is now the place to go to find the authors notes and all sorts of other goodies. AdeleneDawner has kept an archive of Author’s Notes. (This goes up to the notes for chapter 76, and is now not updating. The authors notes from chapter 77 onwards are on hpmor.com.)
The first 5 discussion threads are on the main page under the harry_potter tag. Threads 6 and on (including this one) are in the discussion section using its separate tag system. Also: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16.
As a reminder, it’s often useful to start your comment by indicating which chapter you are commenting on.
Spoiler Warning: this thread is full of spoilers. With few exceptions, spoilers for MOR and canon are fair game to post, without warning or rot13. More specifically: