I think the line of research being conducted by Haidt and others on the psychology of morality is relevant here. His TED talk provides a simple summary: http://www.ted.com/talks/jonathan_haidt_on_the_moral_mind.html
The basic idea is that there are "primary colours" of our moral intuitions that include fairness, harm reduction, loyalty, purity, respect for authority. These considerations give you a web of intuitions about goodness and propriety that are often brought into conflict. You might have a respect for authority, but also recognise that this can cause grave harm. You might have a respect for purity that gives you a knee jerk aversion to promiscuity or homosexuality, but also see that this is unfair to people who have immutable preferences and who aren't harming anyone.
There's no way to reconcile these conflicting intuitions, so we tend to focus on a "prime directive" like "make the world a better place", or "be a fine upstanding individual in your own life and try not to judge other people too harshly".
I think the former, "make the world a better place", is particularly good for producing moral systems that are internally consistent. It does ask you to question and reject a lot of knee jerk reactions. Counter-factuals are useful in doing this. Someone who has trouble letting go of a moral distaste for promiscuity or gluttony should keep in mind that the same instinct could under different circumstances make you repulsed by the idea of a woman on her period preparing food. Someone who thinks there's a moral failure in refusing to stick together with your in-group under adversity should remember that this instinct could have put you on the wrong side of the Holocaust.
Someone who thinks there's a moral failure in refusing to stick together with your in-group under adversity should remember that this instinct could have put you on the wrong side of the Holocaust
From a strictly utilitarian standpoint, if one had a strong commitment to the common good, but uncommonly little knee-jerk reaction or natural empathy, would it have made more sense to passively tolerate the Holocaust/offer only safe resistance, and live to affect the post-war world, where there could be more one could do for oneself/humanity?
Or make a stand an...
It appears to me that much of human moral philosophical reasoning consists of trying to find a small set of principles that fit one’s strongest moral intuitions, and then explaining away or ignoring the intuitions that do not fit those principles. For those who find such moral systems attractive, they seem to have the power of actually reducing the strength of, or totally eliminating those conflicting intuitions.
In Fake Utility Functions, Eliezer described an extreme version of this, the One Great Moral Principle, or Amazingly Simple Utility Function, and suggested that he was partly responsible for this phenomenon by using the word “supergoal” while describing Friendly AI. But it seems to me this kind of simplification-as-moral-philosophy has a history much older than FAI.
For example, hedonism holds that morality consists of maximizing pleasure and minimizing pain, utilitarianism holds that everyone should have equal weight in one’s morality, and egoism holds that moralist consists of satisfying one’s self-interest. None of these fits all of my moral intuitions, but each does explain many of them. The puzzle this post presents is: why do we have a tendency to accept moral philosophies that do not fit all of our existing values? Why do we find it natural or attractive to simplify our moral intuitions?
Here’s my idea: we have a heuristic that in effect says, if many related beliefs or intuitions all fit a certain pattern or logical structure, but a few don’t, the ones that don’t fit are probably caused by cognitive errors and should be dropped and regenerated from the underlying pattern or structure.
As an example where this heuristic is working as intended, consider that your intuitive estimates of the relative sizes of various geometric figures probably roughly fit the mathematical concept of “area”, in the sense that if one figure has a greater area than another, you’re likely to intuitively judge that it’s bigger than the other. If someone points out this structure in your intuitions, and then you notice that in a few cases your intuitions differ from the math, you’re likely to find that a good reason to change those intuitions.
I think this idea can explain why different people end up believing in different moral philosophies. For example, many members of this community are divided along utilitarian/egoist lines. Why should that be the case? The theory I proposed suggests two possible answers:
I think it’s likely that both of these are factors that contribute to the apparent divergence in human moral reasoning. This seems to be another piece of bad news for the prospect of CEV, unless there are stronger converging influences in human moral reasoning that (in the limit of reflective equilibrium) can counteract these diverging tendencies.