I preferred the superhappy ending, and in fact the story nudged me further in that direction. I guess I don't really get what the big deal about pain and suffering is, there's no physical pain on the internet and it seems to work just fine.
three worlds collide would make a decent movie...just have to make the reasoning of the characters more explicit for people unfamiliar with concepts involved.
Eliezer, one qualm: You consistently bring up mirror neurons and consider it to be obvious prima facie that they are used for action understanding in humans. Unfortunately, most contemporary neuroscientists in the field agree that there is no consistent evidence of this:
http://talkingbrains.blogspot.com/2008/08/eight-problems-for-mirror-neuron-theory.html
That is not to say that humans don't understand other people's actions or that we do not have adequate theory of minds! But it does mean that there is no reason to suspect that those complicated cognitive events can be reduced to simply a group of "mirror" neurons. Ramachandran often mentions them as well, which irks me slightly as well.
3WC would be a terrible movie. "There's too much dialogue and not enough sex and explosions", they would say, and they'd be right. And you shouldn't just tack them on, either; sex and explosions should flow out naturally as an indispensable part of the plot.
Andy, consider "mirror neurons" as shorthand for "empathic architecture" rather than implying that the whole thing gets done by a small group of actual neurons a la the "grandmother neuron".
"There's too much dialogue and not enough sex and explosions", they would say, and they'd be right. And you shouldn't just tack them on, either; sex and explosions should flow out naturally as an indispensable part of the plot.
Bullshit. There's that part about rape having been legalized, and the canonical ending involves a planet being destroyed by a supernova. In a visual medium, the hypothetical discussion about exterminating baby-eaters would be voiceover for a montage of helpless crystalline civilians being hunted down by human infantry with power armor and personal laser cannons, culminating in an orbital bombardment.
Eliezer: if you show more of it from the perspective of the Superhappies, dialogue itself takes care of that problem.
"3WC would be a terrible movie. "There's too much dialogue and not enough sex and explosions", they would say, and they'd be right."
Hmmm.. Maybe we should put together a play version of 3WC; plays can't have sex and explosions in any real sense, and dialogue is a much larger driver.
And moreover, I had gone to considerable length to present the Superhappy argument in the best possible light.
Hmm. I felt that while the Superhappy argument was presented in the best possible light, the Superhappy ending wasn't. The non-Superhappy ending was in two parts and contained all kinds of cool things, like the use of emergency flags to manipulate the local prediction markets and more insights into the personalities of the different characters. The Superhappy ending, on the other hand, was just one part and was basically just a pretty dull overview of how everybody considered humanity's future to be horrible.
To me, it felt like a pretty blatant statement of "this is the future that we don't want".
Frankly, what I've kinda on and off wanted to see was someone turn Nick Bostrom's "Fable of the Dragon Tyrant" into a movie. That could, perhaps, actually work. Maybe.
I think that you make good points about how fiction can be part of a valid moral argument, perhaps even an indispensable part for those who haven't had some morally-relevant experience first-hand.
But I'm having a hard time seeing how your last story helped you in this way. Although I enjoyed the story very much, I don't think that your didactic purposes are well-served by it.
My first concern is that your story will actually serve as a counter-argument for rationality to many readers. Since I'm one of those who disagreed with the characters' choice to des...
Stories and movies are deeply different media. I'm surprised the transition works as often as it does.
I've recently become jaded on the use of fiction due to a nearly opposite line of thought: Fiction is really not important.
There are plenty of real problems and real drama.
Eliezer was not able to write about the experience of being a rationalist merely because he read about the experience of being a rationalist in some other work of fiction. Narrative might be required to convey experience, fiction is not.
It is the truths you are trying to convey, that is what matters. But to this we add invented societies, speculation about future technologies, or future ...
"Without that ability to sympathize, we might think that it was perfectly all right* to keep slaves."
Nearly all people for thousands of years thought it was perfectly all right to keep slaves. Are you saying they didn't have the ability to sympathize? This is the sort of profoundly ahistorical "thinking" that irritates so many people. Someone who considers his own society's beliefs to be laws of reality when there is obvious historical evidence in the other direction that they never bothered to think about.
"The complex and detailed universes inevitably lead to utterly pointless arguments."
This is actually an argument for using fiction. Real situations are more complex and much, much more likely to result in peripheral arguments than fictional situations.
@billswift
"Nearly all people for thousands of years thought it was perfectly all right to keep slaves"
And many still do today - for example, Shari'a endorses slavery. Our Western values are far from universal and cannot be taken for granted.
Tom: "Hmmm.. Maybe we should put together a play version of 3WC [...]"
That reminds me! Did anyone ever get a copy of the script to Yudkowski Returns? We could put on a benefit performance for SIAI!
ducks
And perhaps the current slavers would change their minds if they read the right book, and perhaps not - more probably not, I think, without other changes as well. As I noted in the post text, the mirror neurons do have an off switch. It might take some abstract argument to turn them back on. Or it might take a "slave" rescuing their daughter in real life, instead of fiction. Maybe even that wouldn't do it. Maybe their and my reflective equilibria are so far apart that they can't be called by the same word "right".
Nonetheless - Uncl...
Tyrrell said:
Nor did I see anything in your preceding "rigorous" posts to establish that being modified fell in this range It appeared to be a moral assertion for which no argument was given.
Yeah. Eliezer, in your story, being modified just didn't seem bad enough to be obviously preferable to killing 15 billion people. This creates moral ambiguity that is great in a story, but not if you wanted to communicate a clear moral.
The way the story was presented, I was think "humanity without suffering, and having to eat non-sentient babies?... Is that really so bad to justify killing 15 billion people?" Now, as a reader of Overcoming Bias, I know that Value is Fragile, and that scaling up the human brain is a highly risky proposition that that the brain is not designed for. So, the end result of the Superhappy proposal would not be "Humans minus suffering, eating non-sentient babies." It would not be human at all.
Superhappies can't just surgically remove negative emotions and pain from our brains and leave everything else untouched. It would be likely the Superhappies would make a first pass and remove suffering, but the neurochemical changes would drive ...
billswift: But those peripheral argument will still be about things that in some sense matter, as opposed to say, midichlorians.
HughRistik:
Speaking as a new reader of Overcoming Bias myself--I think that the sort of people who read this blog are more likely to miss how dangerous the Superhappies are, because we've considered ways that human suffering could be reduced or eliminated while still letting humans develop properly. Then, when people who already have ideas about how to reduce suffering read that the Superhappies want to eliminate suffering, they assume that the Superhappies' plans are the same as their own. (I'm not sure if this is a previously discussed and named bias, bu...
Uncle Tom's Cabin is not a valid argument that slavery is wrong. "My mirror neurons make me sympathize with a person whose suffering is caused by Policy X" to "Policy X is immoral and must be stopped" is not a valid pattern of inference.
Consider a book about the life of a young girl who works in a sweatshop. She's plucked out of a carefree childhood, tyrannized and abused by greedy bosses, and eventually dies of work-related injuries incurred because it wasn't cost-effective to prevent them. I'm sure this book exists, though I haven't personally come across it. And I'm sure this book would provide just as emotionally compelling an argument for banning sweatshops as Uncle Tom's Cabin did for banning slavery.
But the sweatshop issue is a whole lot more complex than that, right? And the arguments in favor of sweatshops are more difficult to put into novel form, or less popular among the people who write novels, or simply not mentioned in that particular book, or all three.
The problem with fiction as evidence is that it's like the guy who say "It was negative thirty degrees last night, worst snowstorm in fifty years, so how come them liberals are still talking a...
Not enough sex and explosions in 3WC? Are you joking?
Oh, and it would be easier to find someone make it into a good visual novel rather than a good movie.
@Yvain: Mathematical proof is a valid argument, even if it doesn't contain any information, what was true remains so.
Fiction isn't supposed to act as evidence, it's supposed to place you in a specific focus of attention, where you resolve your own questions for yourself, from evidence you already hold. It doesn't explicitly state abstractions which you are supposed to learn in order to master new thoughts, reinterpret old data, or bind existing morals. It invites you to invent abstractions on a given topic for yourself.
Of course, all the usual biases will ...
Just on the off-chance, are there any OB readers who could get a good movie made?
If machinima counts as "a good movie" you might want to talk to Hugh Hancock (I've no idea if he reads OB, but based on his other interests he may well do).
Eliezer, as I indicate in my new post, the issue isn't so much whether you the author judge that some fiction would help inform readers about morals, but whether typical readers can reasonably trust your judgment in such things, relative to the average propaganda content of authors writing apparently similar moral-quandary stories.
I know that Tucker Max, whose movie I Hope They Serve Beer in Hell will be out sometime this year, has read this site since July of 2007 at least. He's actually how I discovered Overcoming Bias.
He's said numerous times that Eliezer would be absolutely fantastic if his posts weren't so ridiculously long and wandering at times.
He'd be a good person to talk to about making a movie (since his own was designed specifically to avoid that "dumbing-down process") and is probably going to make several tens of millions over the next few years.
Why should we believe there are "moral truths"? And why are the rules so different with regard to physics? What other topics have a standard more like morality than physics?
I agree with Yvain. The mirror neuron argument was just shoddy. After acknowledging that the science didn't necessarily support your point about them, you then said that doesn't matter. If the truth of an argument is irrelevant, why bring it up at all? Doesn't such an argument falling back on "deeper truth" have the same weaknesses as the religious/mystical in their ...
"But those peripheral argument will still be about things that in some sense matter, as opposed to say, midichlorians."
Of course they are. That's why they distract so strongly from the central point.
Why would anyone argue about midi-chlorians? The "explanation" of Jedi powers in that movie detracted from the Star Wars universe.
Of course they are. That's why they distract so strongly from the central point.
Even if they distract more from the central point, theya re still real. They still have some potential relevance to the reader.
I think it would be a disservice to train someone up in the arts of rationality only to have most of their thoughts revolve around the facts of some fantasy universe.
Also, just because our universe is more detailed than fictional ones, doesn't mean that all of that detail has to be available to the reader. We can offer simplified descriptions of situations.
Eliezer: It may be worth noting that SIAI just hired a new president FROM a branch of the film industry who has some familiarity with the sort of tax laws that can make indie movies a good investment even when expected value appears negative, and that SIAI's largest donor is the producer of an excellent movie about the marketing of cigarettes.
Other than that.
I agree with Kaj I really like Hugh's point I don't think 3WC or Dragon Tyrant work as movies. I don't know what Eliezer's got however WRT stories.
Tree Frog: do you know Tucker and are you suggesting that I speak with him? That's basically my job after all.
I still don't see the actual practical benefit of suffering. I've lived a very sheltered life, physically and emotionally. I've never needed stitches, never had my heart broken, I've always been pretty low-key emotionally, and I don't feel like I'm missing anything.
Besides, what are we going to do NEXT time we run into a more advanced race of aliens? I suppose we can just keep blowing up starlines, but what happens if they get the jump on us, like the superhappies got the jump on the babyeaters? It seems like we need powerful allies much more than we need our precious aches and pains.
Eliezer,
I am very much inclined to analyze your articles because you are indeed very enthusiastic about your theories, which is a rarity these days. On the link there’s a Wordle tag cloud picture of your article.
As you can see, there’s a lot of “argument(s)”,”moral”, “abstract”, “fiction’, and a somewhat humble “experience”.
To the point – fiction in the literary domain is often a method for implying moral concepts. But fiction is, above all, an invitation to imagine. There is a catch. We can imagine a setting, a world, a relationship. But we sometimes cann...
@Robin: Thank you. Somehow I missed that post, and it was exactly what I was looking for.
@Vladimir Nesov: I agree with everything you said except for your statement that fiction is a valid argument, and your supporting analogy to mathematical proof.
Maybe the problem is the two different meanings of "valid argument". First, the formal meaning where a valid argument is one in which premises are arranged correctly to prove a conclusion eg mathematical proofs and Aristotelian syllogisms. Well-crafted policy arguments, cost-benefit analyses, and stati...
Well, my point is that fiction isn't argument at all, it's a theme for musing on your own questions. It's sometimes useful to read even something you know to be wrong, on a theme interesting for you, written by a thoughtful author. You don't expect to move towards agreement, you know the stuff is wrong, but you can light sparks of your own insight off its pages. When given a mathematical proof, its correctness is for you to appraise. Fiction gives you your own thoughts, take them or leave them.
Michael Vassar: I have no idea who you are, but I'll proceed on the assumption that the "job" you mention is one of representing Eliezer and/or the other OvercomingBias authors in some sort of business capacity.
I don't know Max on a personal level. We've talked a few times on his board and he might be dimly aware of my existence, but I make no claims as to what he will do if contacted by Eliezer or an agent of Eliezer's.
Serious discussions of potential OvercomingBias projects/movies and whatnot should be sent to Max's assistant, Ian Claudius - ian.claudius(AT)gmail.com. The Rudius people are smart and good content creators (multiple book contracts, one soon to be hit movie and stuff I actually like).
Whatever the potential abuses, I think fiction has a valid role to play in dealing with philosophical questions.
One example that comes to mind: I've observed in several discussions on the problem of evil that theists tend to want to discuss the matter in the most abstract possible terms. It seems to be easier to swallow a theodicy when you don't have the unpleasant facts of extreme suffering vividly in mind.
In the case of the POE, fiction can serve to cut through rationalizations in a way argumentation alone can never hope to do.
Eliezer,
Took a bit after reading your babyeater pieces to get my thoughts in order, but my general picture is that you're mis-representing the human condition, and that the whole of the story relies upon that misrepresentation. This is humanity you're talking about. While the readership of OB may match your profile decently, the human species, even extended into a moderately improved state does not hold the value-set you represent.
Child-love/protection is (a) proximity-focused, (b) stronger than abstract value of reciprocity. Folks are much better at
I think I have a co-operation instinct that is pushing me towards the supper happy future.
It feels better, but is probably not what I would do In real life. or I am more different then others then I give credit for.
Telling people about the logical fallacy of generalization from fictional evidence doesn't make it go away; people may just go on generalizing from the story as though they had actually seen it happen.
That danger can be minimized in an elegant way: by referring people to a story that shows one shouldn't generalize from fictional evidence.
The more likely they are to inappropriately take the messages of stories to heart, the more they will be to take to heart the message not to do that.
I actually did try to send a long message consisting only of
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
Reading that was quite effective in pushing me towards Torture.
(I'm not being sarcastic.)
A comment by Anonymous on Three Worlds Collide:
After reading this story I feel myself agreeing with Eliezer more on his views and that seems to be a sign of manipulation and not of rationality.
Philosophy expressed in form of fiction seems to have a very strong effect on people - even if the fiction isn't very good (ref. Ayn Rand).
Robin has similar qualms:
Since people are inconsistent but reluctant to admit that fact, their moral beliefs can be influenced by which moral dilemmas they consider in what order, especially when written by a good writer. I expect Eliezer chose his dilemmas in order to move readers toward his preferred moral beliefs, but why should I expect those are better moral beliefs than those of all the other authors of fictional moral dilemmas?
If I'm going to read a literature that might influence my moral beliefs, I'd rather read professional philosophers and other academics making more explicit arguments.
I replied that I had taken considerable pains to set out the explicit arguments before daring to publish the story. And moreover, I had gone to considerable length to present the Superhappy argument in the best possible light. (The opposing viewpoint is the counterpart of the villain; you want it to look as reasonable as possible for purposes of dramatic conflict, the same principle whereby Frodo confronts the Dark Lord Sauron rather than a cockroach.)
Robin didn't find this convincing:
I don't think readers should much let down their guard against communication modes where sneaky persuasion is more feasible simply because the author has made some more explicit arguments elsewhere... Academic philosophy offers exemplary formats and styles for low-sneak ways to argue about values.
I think that this understates the power and utility of fiction. I once read a book that was called something like "How to Read" (no, not "How to Read a Book") which said that nonfiction was about communicating knowledge, while fiction was about communicating experience.
If I want to communicate something about the experience of being a rationalist, I can best do it by writing a short story with a rationalist character. Not only would identical abstract statements about proper responses have less impact, they wouldn't even communicate the same thought.
From The Failures of Eld Science:
"...Work expands to fill the time allotted, as the saying goes. But people can think important thoughts in far less than thirty years, if they expect speed of themselves." Jeffreyssai suddenly slammed down a hand on the arm of Brennan's chair. "How long do you have to dodge a thrown knife?"
"Very little time, sensei!"
"Less than a second! Two opponents are attacking you! How long do you have to guess who's more dangerous?"
"Less than a second, sensei!"
"The two opponents have split up and are attacking two of your girlfriends! How long do you have to decide which one you truly love?"
"Less than a second, sensei!"
"A new argument shows your precious theory is flawed! How long does it take you to change your mind?"
"Less than a second, sensei!"
"WRONG! DON'T GIVE ME THE WRONG ANSWER JUST BECAUSE IT FITS A CONVENIENT PATTERN AND I SEEM TO EXPECT IT OF YOU! How long does it really take, Brennan?"
Sweat was forming on Brennan's back, but he stopped and actually thought about it -
"ANSWER, BRENNAN!"
"No sensei! I'm not finished thinking sensei! An answer would be premature! Sensei!"
"Very good! Continue! But don't take thirty years!"
This is an experience about how to avoid completing the pattern when the pattern happens to be blatantly wrong, and how to think quickly without thinking too quickly.
Forget the question of whether you can write the equivalent abstract argument that communicates the same thought in less space. Can you do it at all? Is there any series of abstract arguments that creates the same learning experience in the reader? Entering a series of believed propositions into your belief pool is not the same as feeling yourself in someone else's shoes, and reacting to the experience, and forming an experiential skill-memory of how to do it next time.
And it seems to me that to communicate experience is a valid form of moral argument as well.
Uncle Tom's Cabin was not just a historically powerful argument against slavery, it was a valid argument against slavery. If human beings were constructed without mirror neurons, if we didn't hurt when we see a nonenemy hurting, then we would exist in the reference frame of a different morality, and we would decide what to do by asking a different question, "What should* we do?" Without that ability to sympathize, we might think that it was perfectly all right* to keep slaves. (See Inseparably Right and No License To Be Human.)
Putting someone into the shoes of a slave and letting their mirror neurons feel the suffering of a husband separated from a wife, a mother separated from a child, a man whipped for refusing to whip a fellow slave - it's not just persuasive, it's valid. It fires the mirror neurons that physically implement that part of our moral frame.
I'm sure many have turned against slavery without reading Uncle Tom's Cabin - maybe even due to purely abstract arguments, without ever seeing the carving "Am I Not a Man and a Brother?" But for some people, or for a not-much-different intelligent species, reading Uncle Tom's Cabin might be the only argument that can turn you against slavery. Any amount of abstract argument that didn't fire the experiential mirror neurons, would not activate the part of your implicit should-function that disliked slavery. You would just seem to be making a good profit on something you owned.
Can fiction be abused? Of course. Suppose that blacks had no subjective experiences. Then Uncle Tom's Cabin would have been a lie in a deeper sense than being fictional, and anyone moved by it would have been deceived.
Or to give a more subtle case not involving a direct "lie" of this sort: On the SL4 mailing list, Stuart Armstrong posted an argument against TORTURE in the infamous Torture vs. Dust Specks debate, consisting of a short story describing the fate of the person to be tortured. My reply was that the appropriate counterargument would be 3^^^3 stories about someone getting a dust speck in their eye. I actually did try to send a long message consisting only of
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
for a thousand lines or so, but the mailing software stopped it. (Ideally, I should have created a webpage using Javascript and bignums, that, if run on a sufficiently large computer, would print out exactly 3^^^3 copies of a brief story about someone getting a dust speck in their eye. It probably would have been the world's longest finite webpage. Alas, I lack time for many of my good ideas.)
Then there's the sort of standard polemic used in e.g. Atlas Shrugged (as well as many less famous pieces of science fiction) in which Your Beliefs are put into the minds of strong empowered noble heroes, and the Opposing Beliefs are put into the mouths of evil and contemptible villains, and then the consequences of Their Way are depicted as uniformly disastrous while Your Way offers butterflies and apple pie. That's not even subtle, but it works on people predisposed to hear the message.
But to entirely turn your back on fiction is, I think, taking it too far. Abstract argument can be abused too. In fact, I would say that abstract argument is if anything easier to abuse because it has more degrees of freedom. Which is easier, to say "Slavery is good for the slave", or to write a believable story about slavery benefiting the slave? You can do both, but the second is at least more difficult; your brain is more likely to notice the non-sequiturs when they're played out as a written experience.
Stories may not get us completely into Near mode, but they get us closer into Near mode than abstract argument. If it's words on paper, you can end up believing that you ought to do just about anything. If you're in the shoes of a character encountering the experience, your reactions may be harder to twist.
Contrast a verbal argument against the verbal belief that "non-Catholics go to Hell"; versus reading a story about a good and decent person, who happens to be a Protestant, and dies trying to save a child's life, who is condemned to hell and has molten lead poured down her throat; versus the South Park episode where a crowd of newly dead souls is at the entrance to hell, and the Devil says, "Sorry, it was the Mormons" and everyone goes "Awwwww..."
Yes, abstraction done right can keep you going where concrete visualization breaks down - the torture vs. dust specks thing being an archetypal example; you can't actually visualize that many dust specks, but if you try to choose SPECKS you'll end up with circular preferences. But so far as I can organize my metaethics, the ground level of morality lies in our preferences over particular, concrete situations - and when these can be comprehended as concrete images at all, it's best to visualize them as concretely as possible. Unless we know specifically where the concrete image is going wrong, and have to apply an abstract correction. The moral abstraction is built on top of the ground level.
I am also, of course, worried about the idea that stories aren't "respectable" because they don't look sufficiently solemn and dull; or the idea that something isn't "respectable" if can be understood by a mere popular audience. Yes, there are technical fields that are genuinely impossible to explain to your grandmother in an hour; but ceteris paribus, people who can write at a more popular level without distorting technical reality are performing a huge service to that field. I've heard that Carl Sagan was held in some disrepute by his peers for the crime of speaking to the general public. If true, this is merely stupid.
Explaining things is hard. Explainers need every tool they can get their hands on - as a matter of public interest.
And in moral philosophy - well, I suppose it could be the case that moral philosophers have discovered moral truths that are deductive consequences of most humans' moral frames, but which are so difficult and technical that they simply can't be explained to a popular audience within a one-hour lecture. But it would be a tad more suspicious than the corresponding case in, say, physics.
I realize that I speak as someone who does a lot of popularizing, but even so - fiction ought to be a respectable form of moral argument. And a respectable way of communicating experiences, in particular the experience of applying certain types of thinking skills.
I've always been of two minds about publishing longer fiction pieces about the future and its consequences. Not so much because of the potential for abuse, but because even when not abused, fiction can still bypass critical faculties and end up poured directly into the brains of at least some readers. Telling people about the logical fallacy of generalization from fictional evidence doesn't make it go away; people may just go on generalizing from the story as though they had actually seen it happen. And you simply can't have a story that's a rational projection; it's not just a matter of plot, it's a matter of the story needing to be specific, rather than depicting a state of epistemic uncertainty.
But to make shorter philosophical points? Sure.
And... oh, what the hell. Just on the off-chance, are there any OB readers who could get a good movie made? Either outside Hollywood, or able to bypass the usual dumbing-down process that creates a money-losing flop? The probabilities are infinitesimal, I know, but I thought I'd check.