Why should we want to construct a consistent theory?
There are domains where my moral intuition fails me completely. I don't get to abstain from moral questions because I don't like them, so I need to develop some way of making decisions. I would like to be able to argue with people about what the correct decision is, even and especially on hard problems which aren't instantly solved by my intuitions. To handle this situation it looks like I either need to develop a consistent morality, to develop a logical system which can handle inconsistency, to give up on moral arguments, or to somehow enlarge my moral intuitions so they can act as a sanity check on arguments in all domains.
You also don't have a simple or consistent framework to evaluate aesthetic arguments, but I suppose you don't worry about it too much.
What is an aesthetic argument? I have occasionally engaged in arguments about what a certain audience will appreciate, or what I will appreciate in the future, but those are questions of fact which I evaluate in the same way as other questions of fact. I have occasionally gotten into arguments about what artistic sensibilities should be encouraged, but those are generally moral arguments that happen to deal with artwork (and in this case I do worry about it as much as I worry about morality at all).
Shouldn't we conclude that creating neat consistent prescriptive moral theories is futile?
No theory of everything has been visibly successful so far. I'm not too optimistic about finding a simple moral theory which agree with my intuition in the domain where my intuition is defined,but not because of past failures.
I have occasionally gotten into arguments about what artistic sensibilities should be encouraged, but those are generally moral arguments that happen to deal with artwork (and in this case I do worry about it as much as I worry about morality at all).
Yes, I have had in mind such arguments, and even more simply arguments over what's beautiful. I wouldn't classify them under morality label, but if you do, and are consistently worried about them in the same way as about other moral arguments, fine with me. I appologise for expecting that you are less consi...
It appears to me that much of human moral philosophical reasoning consists of trying to find a small set of principles that fit one’s strongest moral intuitions, and then explaining away or ignoring the intuitions that do not fit those principles. For those who find such moral systems attractive, they seem to have the power of actually reducing the strength of, or totally eliminating those conflicting intuitions.
In Fake Utility Functions, Eliezer described an extreme version of this, the One Great Moral Principle, or Amazingly Simple Utility Function, and suggested that he was partly responsible for this phenomenon by using the word “supergoal” while describing Friendly AI. But it seems to me this kind of simplification-as-moral-philosophy has a history much older than FAI.
For example, hedonism holds that morality consists of maximizing pleasure and minimizing pain, utilitarianism holds that everyone should have equal weight in one’s morality, and egoism holds that moralist consists of satisfying one’s self-interest. None of these fits all of my moral intuitions, but each does explain many of them. The puzzle this post presents is: why do we have a tendency to accept moral philosophies that do not fit all of our existing values? Why do we find it natural or attractive to simplify our moral intuitions?
Here’s my idea: we have a heuristic that in effect says, if many related beliefs or intuitions all fit a certain pattern or logical structure, but a few don’t, the ones that don’t fit are probably caused by cognitive errors and should be dropped and regenerated from the underlying pattern or structure.
As an example where this heuristic is working as intended, consider that your intuitive estimates of the relative sizes of various geometric figures probably roughly fit the mathematical concept of “area”, in the sense that if one figure has a greater area than another, you’re likely to intuitively judge that it’s bigger than the other. If someone points out this structure in your intuitions, and then you notice that in a few cases your intuitions differ from the math, you’re likely to find that a good reason to change those intuitions.
I think this idea can explain why different people end up believing in different moral philosophies. For example, many members of this community are divided along utilitarian/egoist lines. Why should that be the case? The theory I proposed suggests two possible answers:
I think it’s likely that both of these are factors that contribute to the apparent divergence in human moral reasoning. This seems to be another piece of bad news for the prospect of CEV, unless there are stronger converging influences in human moral reasoning that (in the limit of reflective equilibrium) can counteract these diverging tendencies.