Why not put a copy of those acknowledgements at the bottom of the story itself, as well?
I suspect lots of people are going to see the story and only think of it as a neat story. Explicitly bringing out the lesson would help, plus I thought the Passover example was really interesting. It wouldn't hurt to make more people see it.
I wonder if you might have seen this essay by David Brin...
Now ponder something that comes through even the party-line demonization of a crushed enemy -- this clear-cut and undeniable fact: Sauron's army was the one that included every species and race on Middle Earth, including all the despised colors of humanity, and all the lower classes.
Hmm. Did they all leave their homes and march to war thinking, "Oh, goody, let's go serve an evil Dark Lord"?
Or might they instead have thought they were the "good guys," with a justifiable grievance worth fighting for, rebelling against an ancient, rigid, pyramid-shaped, feudal hierarchy topped by invader-alien elfs and their Numenorean-colonialist human lackeys?
Picture, for a moment, Sauron the Eternal Rebel, relentlessly maligned by the victors of the War of the Ring -- the royalists who control the bards and scribes (and moviemakers). Sauron, champion of the common Middle Earthling! Vanquished but still revered by the innumerable poor and oppressed who sit in their squalid huts, wary of the royal secret police with their magical spy-eyes, yet continuing to whisper stories, secretly dreaming and hoping that someday he will return ... bringing more rings.
My guess would be that Mordor is a totalitarian communist state, formed on promises of empowerment of the People, and then turned into a horrible labor camp with collective farms by lake Nurnen and armies of expendable mooks kept in line by harsh superiors (think Commissars), along with heavy racist and nationalistic propaganda so they hate their enemies more than they hate their own rulers. Remember the communist revolution that happened in the Shire while our heroes were out destroying the One Ring? It started out as a ham-fisted attempt at social justice, and before long people were disappearing for being enemies of the state. Imagine that, but on a larger scale, and many times worse, and festering for generations.
The orcs (et al) don't have to be inherently evil for Sauronland to be an evil nation.
Apartheid based on age that replaces the previous versions based on race and sex.
The morally indefensible and insanely self-destructive attempt to mitigate drug addiction by banning drugs.
What misdeeds are we shrugging off because they're normal?
Religion. Schools. Television. Not caring about people remote from you. Spending effort on trifles. Akrasia. Irrationality.
Some would say, having political beliefs different from mine.
Burial/cremation.
Loss of time to work. Loss of utility to unemployment.
The way children get so few civil rights they're used as excuses for removing rights from adults.
The way children get so few civil rights they're used as excuses for removing rights from adults.
I am aware of the poor state of affairs re: children's rights, but I'm not sure what you're getting at by citing consequences for adults. Can you elaborate?
Just think how much legislation that restricts adults has been sold on the premise that it "protects children", especially from non-harmful things like porn and homosexuality.
having to kowtow/kiss-up to bullies because they're part of a hierarchy (eg in business), rather than being treated with respect as a human being. Further - being expected to also treat your subordinates badly and thus perpetuate the hierarchy.
What misdeeds are we shrugging off because they're normal?
Eating mammals. More generally; non-vegetarianism.
You missed the step where you assert that most people assert it is wrong to eat sentient animals, which is what would create the inconsistency, were most people to assert that.
You also need the word sentient to mean the same thing in both premises, otherwise it's like “feathers are light, what's light is not dark, therefore feathers are not dark”.
Actually, "Would you eat a Wookie?" is probably a helpful distinguishing question here. For me the answer is obviously "No!" and occurs with the same fleeting nausea as "Would you eat a human being?" But I grew up reading SF books like Little Fuzzy that teach personhood theory in a very visceral way. Other readers claimed they weren't bothered by the Babyeaters because the children eaten weren't human!
because the children eaten weren't human!
Indeed, one thing that surprises ethicists their first time teaching is that in ordinary English, 'person' and 'human' mean the same thing - so most intro students, when asked 'is Yoda a person' will answer 'no', even though they'd answer 'yes' to 'is Luke Skywalker a person'.
I was just the other day lamenting how many people, even largely intelligent and conscientious seeming individuals, answer "Yes" to the OkCupid match question "If you landed on an alien planet where the local intelligent life form tasted unbelievably good, would you eat them?"
I'm TAing discussion sections for the first time today, and based on some of the nonsense the students spouted in lecture yesterday, I'm going to need to cover what those words mean.
I imagine that would be because most people don't understand that sentient beings includes chickens, lobsters[1], and unborn fetuses (not that many people would agree with eating fetuses). If you asked "is that because you think it's wrong to kill and eat beings that are capable of perceiving stimuli" most would probably disagree with you. Now, if you asked "is that because you think it's wrong to kill and eat beings that are capable of doing algebra," you'd probably get a different response.
The reason people wouldn't eat an elf isn't because it's a sentient being, it's because it's a human equivalent sentient being. So you need to reach beyond sentience to find your inconsistency.
And of course, the reason people wouldn't eat a Wookie is because it probably would taste like an old boot.
[1]Research in recent years suggests that crustaceans may be capable of feeling pain and stress.
Part of what bothers me about the idea of eating an elf or Wookie is that they don't feel like prey -- they feel like peers. When I see a fox, e.g., it doesn't make me hungry -- the fox doesn't seem like it's below me on the food chain. When I see a rabbit or a pigeon, it does make me hungry -- I can imagine what it would be like to hunt, clean, roast, and gnaw on it.
On the other hand, I wouldn't hesitate to kill 5 foxes to save one elf or human or Wookie.
I would not, however, hunt a rabbit or a fox for sport; that seems unnecessarily cruel.
One way of accounting for all these moral intuitions is that rabbits, foxes, and Wookies are all sentient; one should not cause pain to sentient creatures for amusement. Foxes and Wookies are ecological peers; one should not eat ecological peers. Wookies are people; one should not trade off the lives of people against roughly comparable numbers of lives of non-people.
Specifically, most people assert that animals are sentient; yet most people are not vegetarians, even though eating meat is no longer necessary for survival. There is an inconsistency between these positions.
No there isn't. It implies that they violate another norm that you value but it is not inconsistent.
I don't see why. For clarity, since we probably agree it's wrong, imagine you're making the same argument for cannibalism instead. One person says, "I'm fine with eating and farming humans but if I get to know one first, doing it would make me feel bad." Another says, "Screw that, I'll eat anyone, even if I know them and their children!"
The second person is more morally consistent and also more callous. Even if there's no difference in the way they live their lives, trying to end the holocaust of humans for food would be easier in a world full of the first type than the second.
Just as I would prefer the opposite of rule of law when the law is uniformly terrible, I prefer the opposite of moral consistency when a morality is terrible.
My first impression of this story was very positive, but as it asks us to ask moral questions about the situation, I find myself doing so and having serious doubts about the moral choices offered.
First of all, it appears to be a choice between two evils, not evil and good. On one hand is a repressive king-based classist society that is undeniably based on socially evil underpinnings. On the other hand we have an absolute unquestionably tyranny that plans to do good. Does no one else have trouble deciding which is the lesser problem?
Secondly, we know for a fact that, in our world, kingdoms and repressive regimes sometimes give way to more enlightened states, and we don't know enough about the world to even know how many different kingdoms there are or what states of enlightenment exist elsewhere. For all we know things are on the verge of a (natural) revolution. We can't say much about rule by an infinite power, having no examples to hand, but there is the statement that "power corrupts". Now, I'm not going to say that this is inevitable, but I have at least to wonder if an integration over total sentient happiness going forward is higher in the old regime and its successo...
Huh; I thought my browser had failed, and this post hadn't appeared. Anyway...
There's an old army saying: "Being in the army ruins action movies for you." I feel the same way about 'scifi' - Aside from season 3, every episode of Torchwood (that I've recently started watching, now that I finished Sopranos) is driving me up the wall. I propose a corollary saying:
"Understanding philosophical materialism and the implications thereof ruins 99% of Science Fiction... and don't get me started on Fantasy!"
In my opinion, there are three essential rules to Fantasy:
The protagonist is a priori important; by their very nature they have metaphysical relevance (even though they don't know it yet!). All other characters are living their rightful and deserved life, unless they are below their means with a Heart of Gold.
The scientific method (hypothesis, experiment, conclusion, theory) only works in the immediate sense, not the broad sense; your immediate world will be logical, but the world as a whole is incomprehensible. You can only build machines if a) they already exist; or b) they serve no practical purpose. Magic, on the other hand, generally works as intended; t
Good story. I love Yudkowskian fiction!
That said, I don't see why Vhazhar would want to touch the Sword of Good if all it does is "test good intentions". Isn't that just another way of saying that if the holder survives, he knows that his terminal values correspond to the Sword's?
It makes sense that Hirou would want Vhazhar to touch the Sword, because since Hirou can touch it, if Vhazhar can touch it too Hirou will know that Vhazhar's terminal values are similar to his own. But why does Vhazhar give a crap about the Sword's terminal values?
If the stories of previous wielders of the Sword were public and reasonably accurate, he presumably already evaluated whether the Sword's terminal values match the terminal values he wished to uphold.
No, the Spell of Infinite Doom destroys the Equilibrium. Light and dark, summer and winter, luck and misfortune - the great Balance of Nature will be, not upset, but annihilated utterly; and in it, set in place a single will, the will of the Lord of Dark. And he shall rule, not only the people, but the very fabric of the World itself, until the end of days.
No matter how good a person the Lord may be, if he's human, I'd have tried to stop the spell.
go for it themselves
What, you mean try to self-modify? Oh hell no. Human brain not designed for that. But you would have a longer time to try to solve FAI. You could maybe try a few non-self-modifications if you could find volunteers, but uploading and upload-driven-upgrading is fundamentally a race between how smart you get and how insane you get.
The modified people can be quite a bit smarter than you are too, so long as you can see their minds and modify them. Groves et al managed to mostly control the Manhattan project despite dozens of its scientists being smarter than any of their supervisors and many having communist sympathies. If he actually shared their earlier memories and could look inside their heads... There's a limit to control, you still won't control an adversarial super intelligence this way, but a friendly human who appreciates your need for power over them? I bet they can have a >50 IQ point advantage, maybe even >70. Schoolteachers control children who have 70 IQ points on them with the help of institutions.
Schoolteachers control children who have 70 IQ points on them with the help of institutions.
Is it relevant that IQ is correlated with obedience to authority?
And how dumb do you think schoolteachers are? Bottom of those with BAs. I'd guess 100. And correlated with their pupils.
Eliezer, I'm with you that a properly designed mind will be great, but mere uploads will still be much more awesome than normal humans on fast forward.
Without hacking on how your mind fundamentally works, it seems pretty likely that being software would allow a better interface with other software than mouse, keyboard and display does now. Hacking on just the interface would (it seems to me) lead to improvements in mental capability beyond mere speed. This sounds like mind hacking to me (software enhancing a software mind will likely lead to blurry edges around which part we call "the mind"), and seems pretty safe.
Some (pretty safe*) cognitive enhancements:
Nope, they didn't get that part wrong.
Look, you should know me well enough by now to know that I don't keep my stories on nice safe moral territory.
A happy ending here is not guaranteed. But think about this very carefully. Are you sure you'd have turned the Sword on Vhazhar? They don't have the same options we do.
He's going to be the emperor. He could implement Parliament, he could create jury trials. He could even put Dolf and Selena on trial for their crimes.
It's interesting that Hirou holds the world accountable to his own moral code, which assumes power corrupts. Then, at the last moment, he grants absolute power to Vhazhar. So in the middle of choosing to use our world's morality, which is built upon centuries of learning to doubt human nature, in the middle of that - Vhazhar's good intentions are so good that they justify granting him absolute power. Lesson not learned.
his own moral code, which assumes power corrupts
Hold on. How can a moral code say anything about questions of fact, such as whether or not power corrupts?
It seems to me that "power corrupts" means "power changes goal content," and that's a purely factual claim.
Well, in fact it would be highly helpful to separate the claims here, even though the factual part is uncontroversial, because it makes it clear what argument is being made, exactly.
And in this case it's uncertain/controversial how much power actually changes behavior, who it changes, how reliably; and this is the key issue, whereas the moral concept that "the behavior of killing everyone who disagrees with you, is wrong" is relatively uncontroversial among us. So calling this a moral claim when the key disputed part is actually a factual claim is a bad idea.
This could be how the 'balance' mythology and the prophecy got started. Perhaps the hero decided long ago that it wasn't worth the risk, and wanted to make sure future heroes kill the Dark Lord.
I assume that the sword tests the correspondence of person's intentions (plan) to their preference.
No, it's the Sword of GOOD. It tests whether you're GOOD, not any of this other stuff.
It should be obvious that the sword doesn't test how well your plans correspond to what you think you want! Otherwise Hirou would have been vaporized.
No, it's the Sword of GOOD. It tests whether you're GOOD, not any of this other stuff.
Wasn't it established that this world's conception of "good" and "evil" are messed up? Why should he trust that the sword really works exactly as advertised?
If the Sword of Good tested whether you're good, Hirou would have been vapourized, because he was obviously not good. He was at the very least an accomplice to murderers, a racist, and a killer.
Well, he was clearly redeemable, at least. It didn't take very much for him to let go of his assumptions, just a few words from someone he thought was an enemy. Making dumb mistakes, even ones with dire consequences, doesn't necessarily make you not Good.
After reading the whole thing, I'm appalled that my only thought against the enforced morality was approximately "they're just worms.." And then immediately accepting the characters' disgust.
FYI, part of the inspiration for this was reading the referenced XKCD and realizing I hadn't gotten that - albeit I first watched The Princess Bride as a child, which may have something to do with it. But yeah, although I seem more resistant to moral dissonance than average - probably more because my mind generally tries to visualize things as real, than out of any innate superethics - I'm still vulnerable to it, and that's part of the horror.
So of course I wanted to share that horror with the rest of you!
The inability to suspend moral disbelief is one of many things that can interfere with the enjoyment of basically good fiction. When I am screaming at characters that they FAIL ETHICS FOREVER, I'm rarely having fun.
Characters who FAIL ETHICS FOREVER can still be entertaining. For example, plenty of villains clearly have little regard for ethics. Authors who FAIL ETHICS FOREVER are usually less desirable. For example, Terry Goodkind. I've rarely felt personally insulted by a work of fiction, but, well, Naked Empire somehow managed to contain the purest, unadulterated essence of Ethics Fail I've ever encountered - it even managed to contradict the explicit moral lessons of the earlier books in the series!
Vfa'g Uvebh'f gehfgvat bs gur Ybeq bs Qnex yvxr gehfgvat gung nal Fvathynevgl jbhyq znxr guvatf orggre? Jung nffhenapr qbrf Uvebh unir gung gur YbQ vf Sevraqyl?
I didn't get that from the story. All those fantasy books he's read, and he only now ponders whether something is good just because the author labeled it "Good"?
I think you're being a little optimistic here in thinking your skepticism is at all general.
Why was Norman Spinrad's _The Iron Dream_ so critically well-received and still read? (If you haven't read it, it's much like Eliezer's story except without the sane hero.) Because it demonstrated that most readers weren't critical, that they'd been reading fantasy stories for literally decades without cottoning onto how well the same stories justified genocide and fascism!
also he seems to be quite an excellent moral philosopher - someone who actually perceives morality.
I didn't get that from the story. All those fantasy books he's read
Not Hirou, Vhazhar. For some reason, even as a very young child facing religious indoctrination, I couldn't quite accept that Abraham had made the right choice in trying to sacrifice Isaac upon God's command. That was one of my first moral breaks with Judaism. The Lord of Dark is - almost necessarily - actually visualizing situations and reacting to them as if seen, rather than processing words however the people around him expect to process them; there's no other way he could reject the values of his society to that extent, and even then, the amount of convergence he exhibits with our own civilization is implausible barring extremely optimistic assumptions about (a) the amount of absolute coherence (b) our own society's intelligence and (c) the Lord of Dark's intelligence; but of course the story wouldn't have worked otherwise.
I guess I'm just surprised to see an allegory from you in which someone solves Friendliness by applying thirty seconds of his at-best-slightly-above-average moral intuition.
Vhazhar's...
Okay, Hirou has evidence that Vhazhar is a moral savant. But the reader, and Hirou, sees little evidence that Vhazhar has worked out a formal, rigorous theory of Friendliness. I thought that anything less than that, on your view, virtually guaranteed the obliteration of almost everything valuable.
But I draw a weaker inference from Vhazhar's ability to overcome indoctrination. Yes, it implies that he probably had a high native aptitude for correct moral reasoning. But the very fact that he was subjected to the indoctrination means that he's probably damaged anyways. If someone survives a disease that's usually deadly, you should expect that she went into the disease with an uncommonly strong constitution. But, given that she's had the disease, you should expect that she's now less healthy than average.
I also didn't consider the possibility that Vhazhar was planning to run the world himself directly. A human just doesn't have the computational capacity to run the world. If a human tried to run the world, there would still be both fortune and misfortune.
There could be less misfortune. A cautious human god who wasn't corrupted by power certainly could plausibly accomplish a lot of good with a few minimal actions. Of course the shaky part is that "cautious" and "not corrupted" part.
I found the ending to be highly telegraphed. No doubt this is partly because I know how the author is likely to think about things, but having the idea of an untrustworthy translation spell introduced in the fourth paragraph, combined with the Excessively Straightforward Names, certainly didn't help. Not to mention the bit about, Oh, let's think about what heroism really entails.
If you meet the buddha on the road, you must kill him.
The koan really strikes me in this situation. The character in the end accepts that he alone gets to make the final moral decisions for himself, regardless of what the labels are and what his teachings were. Many religious ceremonies are about abject submission, but many are also about the idea of "I freely give myself, and accept the consequnces of that submission" and so on.
Although I will add that I completely disagree with the hero. If he really was undecided, the current balance was hi...
Random walks are for agents who have thought through the possibilities and are responding to new information. Hirou's response is far, far more realistic for a human, though perhaps too quick.
I've had similar experiences myself, and try not to have them again. Evidence builds up behind a wall of denial, and when the dam breaks the flood is loosed.
I'd like to think I would have noticed the moral problems with what the "good" guys were doing on my own, and without the benefit of knowing who the author was. I think I would have, but I'm not totally confidence in my Milgram Resistance.
The ending did bother me, though. Why was Hirou willing to believe everything Vhazhar told him without trying to verify it? Why did he kill Dolf instead of accepting that Dolf was simply limited by the moral myopia of his own society, which he clearly was? Maybe exceptionally good people like Vhazhar could see t...
In writing it's even simpler - the author gets to create the whole social universe, and the readers are immersed in the hero's own internal perspective. And so anything the heroes do, which no character notices as wrong, won't be noticed by the readers as unheroic. Genocide, mind-rape, eternal torture, anything.
Not true. If you've got some time to kill, read this thread on The Fanfiction Forum; long story short, a guy who's quite possibly psychopathic writes a story wherein Naruto is turned into a self-centered, hypocritical bastard who happily mindra...
So Chuunin Exam Day, then? I've never read it, but I've heard of it.
Considering that I was able to identify the author and possibly the exact fic from the information that the morality was being heavily lambasted, may I suggest that readers noticing nonlampshaded evil doesn't actually happen all that often? TV Tropes is good at noticing Moral Dissonance, but literally nowhere else that I've ever heard of. It took a critic on the order of David Brin to point out that Aragorn wasn't democratically elected.
Does anyone have, as a step in copyediting, creating a concordance? eg, running "sort | uniq -c"?
Forgive me if this is a stupid question, but is the opening line just a framing device for a short story with abrupt transitions, or does it mean that this is an actual draft of a book that won't be finished for whatever reason?
...and, amazingly enough, FictionPress doesn't allow me to include double spaces in my writing. Deal-breaker in my book, so I'm giving up and hosting on yudkowsky.net instead. Can anyone suggest a better place to post in the future?
is non-breaking. It'll prevent the browser from breaking the line at what ought to be a good place to break it. If you want to force a wider space after a period than the renderer's default, then use .
Hey I know you posted this a long time ago but I found this a few weeks on tvtropes and found it very clever. But the ending confuses me, so I'm hoping you'll reply to my message:
Are we supposed to agree with the antagonist? When I first read it I thought "ahah I get it, its about people blindly accepting whats put in front of them, so Hiro representing that blindly accepted what he said about remaking the world and we're supposed to think about it and realise he was too gullible" then reading some of the comments made me wonder if he was suppose...
I saw what was coming when I got to the bit about the wormarium and Dolf's hyperbolic reaction to it. Are my moral instincts just really warped relative to the norm, or do others agree with me that this was way too obvious?
It's a convention of fantasy and science fiction that there can exist sentient races which are, by their very nature, inimical to mankind, and can therefore be justifiably killed on sight. In principle, there's no reason why such creatures can't actually exist. So that scene didn't set off any alarm bells for me. The first thing that made me look askew was that Hirou's company included both a pirate and a thief, and the wormarium was my confirmation.
Hirou didn't bother to verify that the orcs actually were irredeemably evil, which is possible, but you would have to see evidence. Hirou just saw something physically ugly, and his social surroundings expected him to kill them. But the view which he acted-as-if-believed was possible, it simply wasn't true. Arguably my Orthodox Jewish parents committed more blatant mistakes - if far more theoretical mistakes - in endorsing God's murder of the Egyptian firstborn.
Reading it at first I was sad because I thought it did not belong on Lw but did not want to downvote it. I was happy that the ending justified its inclusion.
Vs Iunmune pna naq jvyy erfgber Nyrx, jul abg Uvebh nf jryy?
[will de-rot13 on request; I don't know what spoiler policy to apply]
In writing it's even simpler - the author gets to create the whole social universe, and the readers are immersed in the hero's own internal perspective. And so anything the heroes do, which no character notices as wrong, won't be noticed by the readers as unheroic. Genocide, mind-rape, eternal torture, anything.
I don't think you give readers enough credit. The author has some influence, but not that much. Some of what appears to be acceptance of the social norms depicted is really just acceptance that the characters live within those norms.
For the i...
I figured out the game that was afoot about a quarter of the way through, but I credit that to the fact that I was trying to write a similar story. Which I may as well abandon now. ;_;
Loved the story and also the first time I took you strong atheism completely seriously, but I think that one bit where they stab those three sleeping guys went a bit too strongly to the "no, this definately isn't right" side of things. Although I didn't think about that scene at all when I was trying to figure out which side was the Good side and thought about the death of Alek as my main piece of evidence for the Lord of Dark being Bad possibility so that's something.
This matches one interpretation of Jack and the Beanstalk, which has been emphasized in some versions of the tale.
At the sentence and paragraph level the story is very well written, but overall I think you'd have made your point better (and it would've fared better as sci-fi) if the ending were more ambiguous. For now it just implies that the fantasy world is American party politics in funny suits: yay to racial equality, boo to the corrupt establishment! The acknowledgements do dispel this impression somewhat, so you could just move them into the story page as Kaj Sotala suggested.
That said, IAWYC completely, and upvoted.
The ending of your story reminds me of the ending of a certain fantasy series, in which the hero manages to successfully gain near-infinite power by using the Extremely Dangerous MacGuffin of Lots of Power, and uses it to create a parallel universe and banish the Bad Guys into it, so they won't be able to go around being Evil at good people any more. It's a damn shame the guy who got to use the MacGuffin was an Objectivist, though. :P
Yeah, Hirou didn't notice the Moral Dissonance...
Once upon a time, and by which I mean about a month ago, I was spending way too much effort trying to understand the criminal known as ziz, and her cult of criminal and non-criminal friends, hanger-ons, and victims.
I bought a $20 wooden fake katana from some random gift shop in Japantown in San Francisco, and started practicing my swings. I bought a bag of oranges, juggled them for a minute, then tried wacking them like my sword to see if it would break. I had to clean diy orange juice off the ceiling.
I probably should have joined a dojo if I actually care...
So I'm noticing what you say about the Seder and having just celebrated passover myself I'd like to offer some perspective that you seem not to have been exposed to.
The Egyptians we're punished because they were equally complicit in our suffering. There is even a commentary that goes so far as to say that there was something of a revolt and pharaoh was originally opposed to enslaving and breaking us but he was ousted from the palace until he capitulated. Further any officer carrying out his orders was blatantly culpable. They had the option of imitating th...
also he seems to be quite an excellent moral philosopher - someone who actually perceives morality.
I didn't get that from the story. All those fantasy books he's read
Not Hirou, Vhazhar. For some reason, even as a very young child facing religious indoctrination, I couldn't quite accept that Abraham had made the right choice in trying to sacrifice Isaac upon God's command. That was one of my first moral breaks with Judaism. The Lord of Dark is - almost necessarily - actually visualizing situations and reacting to them as if seen, rather than processing words however the people around him expect to process them; there's no other way he could reject the values of his society to that extent, and even then, the amount of convergence he exhibits with our own civilization is implausible barring extremely optimistic assumptions about (a) the amount of absolute coherence (b) our own society's intelligence and (c) the Lord of Dark's intelligence; but of course the story wouldn't have worked otherwise.
I guess I'm just surprised to see an allegory from you in which someone solves Friendliness by applying thirty seconds of his at-best-slightly-above-average moral intuition.
Vhazhar's been working on it for some unknown number of years, having successfully realized that sucking the life from worms may be icky but doesn't actually hurt any sentient beings. (Though I wasn't assuming Vhazhar was ancient, he very well could be, and that would make a number of things more plausible, really.) Hirou has a whole civilization behind him and just needed to wake up and actually think.
Okay, Hirou has evidence that Vhazhar is a moral savant. But the reader, and Hirou, sees little evidence that Vhazhar has worked out a formal, rigorous theory of Friendliness. I thought that anything less than that, on your view, virtually guaranteed the obliteration of almost everything valuable.
But I draw a weaker inference from Vhazhar's ability to overcome indoctrination. Yes, it implies that he probably had a high native aptitude for correct moral reasoning. But the very fact that he was subjected to the indoctrination means that he's probably dam...
..fragments of a book that would never be written...
* * *
Captain Selena, late of the pirate ship Nemesis, quietly extended the very tip of her blade around the corner, staring at the tiny reflection on the metal. At once, but still silently, she pulled back the sword; and with her other hand made a complex gesture.
The translation spell told Hirou that the handsigns meant: "Orcs. Seven."
Dolf looked at Hirou. "My Prince," the wizard signed, "do not waste yourself against mundane opponents. Do not draw the Sword of Good as yet. Leave these to Selena."
Hirou's mouth was very dry. He didn't know if the translation spell could understand the difference between wanting to talk and wanting to make gestures; and so Hirou simply nodded.
Not for the first time, the thought occurred to Hirou that if he'd actually known he was going to be transported into a magical universe, informed he was the long-lost heir to the Throne of Bronze, handed the legendary Sword of Good, and told to fight evil, he would have spent less time reading fantasy novels. Joined the army, maybe. Taken fencing lessons, at least. If there was one thing that didn't prepare you for fantasy real life, it was sitting at home reading fantasy fiction.
Dolf and Selena were looking at Hirou, as if waiting for something more.
Oh. That's right. I'm the prince.
Hirou raised a finger and pointed it around the corner, trying to indicate that they should go ahead -
With a sudden burst of motion Selena plunged around the corner, Dolf following hard on her heels, and Hirou, startled and hardly thinking, moving after.
(This story ended up too long for a single LW post, so I put it on yudkowsky.net.
Do read the rest of the story there, before continuing to the Acknowledgments below.)
Acknowledgments:
I had the idea for this story during a conversation with Nick Bostrom and Robin Hanson about an awful little facet of human nature I call "suspension of moral disbelief". The archetypal case in my mind will always be the Passover Seder, watching my parents and family and sometimes friends reciting the Ten Plagues that God is supposed to have visited on Egypt. You take drops from the wine glass - or grape juice in my case - and drip them onto the plate, to symbolize your sadness at God slaughtering the first-born male children of the Egyptians. So the Seder actually points out the awfulness, and yet no one says: "This is wrong; God should not have done that to innocent families in retaliation for the actions of an unelected Pharaoh." I forget when I first realized how horrible that was - the real horror being not the Plagues, of course, since they never happened; the real horror is watching your family not notice that they're swearing allegiance to an evil God in a happy wholesome family Cthulhu-worshiping ceremony. Arbitrarily hideous evils can be wholly concealed by a social atmosphere in which no one is expected to point them out and it would seem awkward and out-of-place to do so.
In writing it's even simpler - the author gets to create the whole social universe, and the readers are immersed in the hero's own internal perspective. And so anything the heroes do, which no character notices as wrong, won't be noticed by the readers as unheroic. Genocide, mind-rape, eternal torture, anything.
Explicit inspiration was taken from this XKCD (warning: spoilers for The Princess Bride), this Boat Crime, and this Monty Python, not to mention that essay by David Brin and the entire Goblins webcomic. This Looking For Group helped inspire the story's title, and everything else flowed downhill from there.
