It seems like a rather different statement to say that there exists a mechanism on our brain which tends to make us want to act as though we had no stakes in the situation, as opposed to talking about what is moral. I'm no evo-psych specialist but it seems plausible that such a mechanism exists. I dispute the notion that such a mechanism encompasses what is usually meant by morality. Most moral systems do not resolve to simply satisfying that mechanism. Also, I see no reason to label that particular mechanism "moral", nor the output of it those things we "should" do (I don't just disagree with this on reflection; it's actually my inuition that "should" means what you want to do, while impartiality is a disconnected preference that I recognize but don't associate even a little bit with should. I don't seem to have an intuition about what morality means other than doing what you should, but then I get a little jarring sensation from the contact with my should intuition...). You've described something I agree with after the taboo, but which before it I definitely disagree with. It's just an issue of semantics at this point, but semantics are also important. "Morality" has really huge connotations for us; it's a bit disingenuous to pick one specific part of our preferences and call it "moral", or what we "should" do (even if that's the part of our brain that causes us to talk about morality, it's not what we mean by morality). I mean, I ignore parts of my preferences all the time. A thousand shards of desire and all that. Acting impartially is somewhere in my preferences,, but it's pretty effectively drowned out by everything else (and I would self-modify away from it given the option - it's not worth giving anything up for on reflection, except as social customs dictate).
I can identify the mechanism you call moral outrage though. I experience (in my introspection of my self-simulation, so, you know, reliable data here /sarcasm) frustration that he would make a decision that would kill me for no reason (although it only just now occurred to me that he could be intentionally evil rather than stupid - that's odd). I oddly experience a much stronger reaction imagining him being an idiot than imagining him directly trying to kill me. Maybe it's a map from how my "should" algorithm is wired (you should do that which on reflection you want to do) onto the situation, which does make sense. I dislike the goals of the evil guy, but he's following them as he should. The stupid one is failing to follow them correctly (and harming me in the process - I don't get anywhere near as upset, although I do get some feeling from it, if he kills 5 to save me).
In short, using the word moral makes your point sound really different than when you don't. I agree with it, mostly, without "moral" or "should". I don't think that most people mean anything close to what you've been using those words to mean, so I recommend some added clarity when talking about it. As to the Squareness Rock, "square" is a useful cocept regardless of how I learned it - and if it was a Harblan Rock that told me a Harblan was a rectangle with sides length 2:9, I wouldn't care (unless there were special properties about Harblans). A Morality Rock only tells me some rules of behavior, which I don't care about at all unless they line up with the preferences I already had. There is no such thing as morality, except in the way it's encoded in individual human brains (if you want to call that morality, since I prefer simply calling it preferences); and your definition doesn't even come close to the entirety of what is encoded in human brains.
By now the Trolley Problem is well known amongst moral philosophers and LessWrong readers. In brief, there's a trolley hurtling down the tracks. The dastardly villain Snidely Whiplash has tied five people to the tracks. You have only seconds to act. You can save the five people by throwing a switch and transferring the trolley to another track. However the evil villain has tied a sixth person to the alternate track. Should you throw the switch?
When first presented with this problem, almost everyone answers yes. Sacrifice the one to save five. It's not a very hard choice.
Now comes the hard question. There is no switch or alternate track. The trolley is still coming down the tracks, and there are still five people tied to it. You are instead standing on a bridge over the tracks. Next to you is a fat man. If you push the man onto the tracks, the trolley car will hit him and derail, saving the five people; but the fat man will die. Do you push him?
This is a really hard problem. Most people say no, they don't push. But really what is the difference here? In both scenarios you are choosing to take one life in order to save five. It's a net gain of four lives. Especially if you call yourself a utilitarian, as many folks here do, how can you not push? If you do push, how will you feel about that choice afterwards?
Try not to Kobayashi Maru this question, at least not yet. I know you can criticize the scenario and find it unrealistic. For instance, you may say you won't push because the man might fight back, and you'd both fall but not till after the trolley had passed so everyone dies. So imagine the fat man in a wheelchair, so he can be lightly rolled off the bridge. And if you're too socially constrained to consider hurting a handicapped person, maybe the five people tied to the tracks are also in wheelchairs. If you think that being pushed off a bridge is more terrifying than being hit by a train, suppose the fat man is thoroughly anesthetized. Yes, this is an unrealistic thought experiment; but please play along for now.
Have your answer? Good. Now comes the third, final, and hardest question; especially for anybody who said they'd push the fat man. There is still no switch or alternate track. The trolley is still coming down the tracks, and there are still five people tied to it. You are still standing on a bridge over the tracks. But this time you're alone and the only way to stop the train is by jumping in front of it yourself. Do you jump? If you said yes, you would push the fat man; but you won't jump. Why?
Do you have a moral obligation to jump in front of the train? If you have a moral obligation to push someone else, don't you have a moral obligation to sacrifice yourself as well? or if you won't sacrifice yourself, how can you justify sacrificing someone else? Is it morally more right to push someone else than jump yourself? I'd argue the opposite...
Realistically you may not be able to bring yourself to jump. It's not exactly a moral decision. You're just not that brave. You accept that it's right for you to jump, and accept that you're not that moral. Fine. Now imagine someone is standing next to you, a skinny athletic person who's too small to stop the train themselves but strong enough to push you over into the path of the trolley. Do you still think the correct answer to the trolley problem is to push?
If we take it seriously, this is a hard problem. The best answer I know is Rawlsianism. You pick your answer in ignorance of who you'll be in the problem. You don't know whether you're the pusher, the pushed, or one of the people tied to the tracks. In this case, the answer is easy: push! There's a 6/7 chance you'll survive so the selfish and utilitarian answers converge.
We can play other variants. For instance, suppose Snidely kidnaps you and says "Tomorrow I'm going to flip a coin. Heads I'll put you on the tracks with 4 other people (and put a different person on the bridge next to the pusher). Tails I'll put you on the bridge next to a pusher." Should the pusher push? Actually that's an easy one because you don't know where you'll end up so you might as well save the four extra people in both scenarios. Your expected value is the same and everyone else's is increased by pushing.
Now imagine Snidely says instead he'll roll a die. If it comes up 1-5, he puts six people including you on the track. If it comes up 6, he lets you go and puts the other five people on the track. However if you agree to be tied to the track without a roll, without even a chance of escape, he'll let the other five people go. What now? Suppose he rolls two dice and they both have to come up 6 for you to go free; but he'll still let everyone else go if you agree. Will you save the other five people at the cost of a 1/36 chance of saving your own life? How about three dice? four? How many dice must Snidely roll before you think the chance of saving your own life is outweighed by the certainty of saving five others?
Do you have your answers? Are you prepared to defend them? Good. Comment away, and you can even Kobayashi Maru the scenario or criticize the excessively contrived hypotheticals I've posed here. But be forewarned, in part 2 I'm going to show you an actual, non-hypothetical scenario where this problem becomes very real; indeed a situation I know many LessWrong readers are facing right now; and yes, it's a matter of life and death.
Update: It now occurs to me that the scenario can be tightened up considerably. Forget the bridge and the fat man. They're irrelevant details. Case 1 is as before. 5 people on one track, 1 on another. Pull the switch to save the 5 and kill the 1. Still not a hard problem.
Case 2: same as before, except this time you are standing next to the one person tied to the track who will be hit by the trolley if you throw the switch. And they are conscious, can talk to you, and see what you're doing. No one else will know what you did. Does this change your answer, and if so why?
Case 3: same as before, except this time you are the one person tied to the track who will be hit by the trolley if you throw the switch.
Folks here are being refreshingly honest. I don't think anyone has yet said they would throw the switch in case 3, and most of us (myself included) are simply admitting we're not that brave/altruistic/suicidal (assuming the five people on the other track are not our friends or family). So let's make it a little easier. Suppose in case 3 someone else, not you, is tied to the track but can reach the switch. What now?
Update 2: Case 4: As in case 3, you are tied to the track, five other unrelated people are tied to the opposite track, and you have access to a switch that will cause the trolley to change tracks. However now the trolley is initially aimed at you. The five people on the other track are safe unless you throw the switch. Is there a difference between throwing the switch in this case, and not throwing the switch in Case 3?
This case also raises the interesting question of legality. If there are any lawyers in the room, do you think a person who throws the switch in case 4--that is, saves themselves at the cost of five other lives--could be convicted of a crime? (Of course, the answer to this one may vary with jurisdiction.) Are there any actual precedents of cases like this?