Because the Sword of Good didn't kill him;
Why does Hirou trust the Sword of Good? How does he know that it's Friendly?
also he seems to be quite an excellent moral philosopher - someone who actually perceives morality.
I didn't get that from the story. All those fantasy books he's read, and he only now ponders whether something is good just because the author labeled it "Good"? He only now considers how immoral the actions of many fantasy heroes would be were they real? I remember being bothered by Aragorn's divine right to lead when I was eight and my Dad was reading Lord of the Rings to me.
As your acknowledgments show, pondering whether it could really be moral to kill "bad guys" so willy-nilly is common in fantasy circles. One of the Austin Powers movies used this to humorous effect with a little vignette about how one of the henchmen killed by Powers had a loving family and had just celebrated his retirement surrounded by loving friends.
Maybe these thoughts never occur to many fantasy readers, but I don't think that we're talking about some vanishingly rare perspicacity here.
And if not him, then who else on the next try?
Maybe someone who's developed a rigorous theory of friendliness :).
I guess I'm just surprised to see an allegory from you in which someone solves Friendliness by applying thirty seconds of his at-best-slightly-above-average moral intuition. I did not get the impression that Hirou was any kind of moral savant. And I had thought that even a moral savant, on your view, couldn't reliably make such a decision in thirty seconds.
I didn't get that from the story. All those fantasy books he's read, and he only now ponders whether something is good just because the author labeled it "Good"?
I think you're being a little optimistic here in thinking your skepticism is at all general.
Why was Norman Spinrad's _The Iron Dream_ so critically well-received and still read? (If you haven't read it, it's much like Eliezer's story except without the sane hero.) Because it demonstrated that most readers weren't critical, that they'd been reading fantasy stories for literally decades ...
..fragments of a book that would never be written...
* * *
Captain Selena, late of the pirate ship Nemesis, quietly extended the very tip of her blade around the corner, staring at the tiny reflection on the metal. At once, but still silently, she pulled back the sword; and with her other hand made a complex gesture.
The translation spell told Hirou that the handsigns meant: "Orcs. Seven."
Dolf looked at Hirou. "My Prince," the wizard signed, "do not waste yourself against mundane opponents. Do not draw the Sword of Good as yet. Leave these to Selena."
Hirou's mouth was very dry. He didn't know if the translation spell could understand the difference between wanting to talk and wanting to make gestures; and so Hirou simply nodded.
Not for the first time, the thought occurred to Hirou that if he'd actually known he was going to be transported into a magical universe, informed he was the long-lost heir to the Throne of Bronze, handed the legendary Sword of Good, and told to fight evil, he would have spent less time reading fantasy novels. Joined the army, maybe. Taken fencing lessons, at least. If there was one thing that didn't prepare you for fantasy real life, it was sitting at home reading fantasy fiction.
Dolf and Selena were looking at Hirou, as if waiting for something more.
Oh. That's right. I'm the prince.
Hirou raised a finger and pointed it around the corner, trying to indicate that they should go ahead -
With a sudden burst of motion Selena plunged around the corner, Dolf following hard on her heels, and Hirou, startled and hardly thinking, moving after.
(This story ended up too long for a single LW post, so I put it on yudkowsky.net.
Do read the rest of the story there, before continuing to the Acknowledgments below.)
Acknowledgments:
I had the idea for this story during a conversation with Nick Bostrom and Robin Hanson about an awful little facet of human nature I call "suspension of moral disbelief". The archetypal case in my mind will always be the Passover Seder, watching my parents and family and sometimes friends reciting the Ten Plagues that God is supposed to have visited on Egypt. You take drops from the wine glass - or grape juice in my case - and drip them onto the plate, to symbolize your sadness at God slaughtering the first-born male children of the Egyptians. So the Seder actually points out the awfulness, and yet no one says: "This is wrong; God should not have done that to innocent families in retaliation for the actions of an unelected Pharaoh." I forget when I first realized how horrible that was - the real horror being not the Plagues, of course, since they never happened; the real horror is watching your family not notice that they're swearing allegiance to an evil God in a happy wholesome family Cthulhu-worshiping ceremony. Arbitrarily hideous evils can be wholly concealed by a social atmosphere in which no one is expected to point them out and it would seem awkward and out-of-place to do so.
In writing it's even simpler - the author gets to create the whole social universe, and the readers are immersed in the hero's own internal perspective. And so anything the heroes do, which no character notices as wrong, won't be noticed by the readers as unheroic. Genocide, mind-rape, eternal torture, anything.
Explicit inspiration was taken from this XKCD (warning: spoilers for The Princess Bride), this Boat Crime, and this Monty Python, not to mention that essay by David Brin and the entire Goblins webcomic. This Looking For Group helped inspire the story's title, and everything else flowed downhill from there.