Today's post, Is Morality Preference? was originally published on 05 July 2008. A summary (taken from the LW wiki):
A dialogue on the idea that morality is a subset of our desires.
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Moral Complexities, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
How about a world in which by sending one person from your planet into slavery you defer the enslavement of the entire earth for 140 years? A world in which alien invaders which outgun us more than Europeans outgunned the tribes in Africa from which many of them took slaves, but who are willing for some reason we can't comprehend to take one person you pick back to the home planet, 70 light years away. But failing your making that choice, they will stay here and at some expense to themselves enslave our entire race and planet?
Can you now imagine a world in which your sending someone in to slavery is not immoral? If so, how does this change in what you can and cannot imagine change your opinion of either the imagination standard or slavery's moral status?
It seems to me most likely source of emotions, feelings, is evolution. We aren't just evolved to run from a sabre tooth tiger, we have a rush of overwhelming fear as the instrumentality of our fleeing effectively. SImilarly, we have evolved, mammals as a whole, not just humans not even just primates, to be "social" animals meaning a tremendously important part of the environment was our group of other mammals. Long before we made the argument that slavery was wrong, we had strong feelings of wanting to resist the things that went along with being enslaved, while apparently we also had power feelings that assisted us in forcing others to do what we wanted.
Given the way emotions probably evolved, I think it does make sense to look to our emotions to guide us in knowing what strategies probably work better than others in interacting with our environment, but it doesn't make sense to expect them to guide us correctly in corner cases, in rare situations in which there would have not been enough pay-off to have evolution do any fine tuning of emotional responses.