What do I mean by "morality isn't logical"? I mean in the same sense that mathematics is logical but literary criticism isn't: the "reasoning" we use to think about morality doesn't resemble logical reasoning. All systems of logic, that I'm aware of, have a concept of proof and a method of verifying with high degree of certainty whether an argument constitutes a proof. As long as the logic is consistent (and we have good reason to think that many of them are), once we verify a proof we can accept its conclusion without worrying that there may be another proof that makes the opposite conclusion. With morality though, we have no such method, and people all the time make moral arguments that can be reversed or called into question by other moral arguments. (Edit: For an example of this, see these posts.)
Without being a system of logic, moral philosophical reasoning likely (or at least plausibly) doesn't have any of the nice properties that a well-constructed system of logic would have, for example, consistency, validity, soundness, or even the more basic property that considering arguments in a different order, or in a different mood, won't cause a person to accept an entirely different set of conclusions. For all we know, somebody trying to reason about a moral concept like "fairness" may just be taking a random walk as they move from one conclusion to another based on moral arguments they encounter or think up.
In a recent post, Eliezer said "morality is logic", by which he seems to mean... well, I'm still not exactly sure what, but one interpretation is that a person's cognition about morality can be described as an algorithm, and that algorithm can be studied using logical reasoning. (Which of course is true, but in that sense both math and literary criticism as well as every other subject of human study would be logic.) In any case, I don't think Eliezer is explicitly claiming that an algorithm-for-thinking-about-morality constitutes an algorithm-for-doing-logic, but I worry that the characterization of "morality is logic" may cause some connotations of "logic" to be inappropriately sneaked into "morality". For example Eliezer seems to (at least at one point) assume that considering moral arguments in a different order won't cause a human to accept an entirely different set of conclusions, and maybe this is why. To fight this potential sneaking of connotations, I suggest that when you see the phrase "morality is logic", remind yourself that morality isn't logical.
By "morality" you seem to mean something like 'the set of judgments about mass wellbeing ordinary untrained humans arrive at when prompted.' This is about like denying the possibility of arithmetic because people systematically make errors in mathematical reasoning. When the Pythagoreans reasoned about numbers, they were not being 'sufficiently careful;' they did not rigorously define what it took for something to be a number or to have a solution, or stipulate exactly what operations are possible; and they did not have a clear notion of the abstract/concrete distinction, or of which of these two domains 'number' should belong to. Quite plausibly, Pythagoreans would arrive at different solutions in some cases based on their state of mind or the problems' framing; and certainly Pythagoreans ran into disagreements they could not resolve and fell into warring camps as a result, e.g., over whether there are irrational numbers.
But the unreasonableness of the disputants, no matter how extreme, cannot infect the subject matter and make that subject matter intrinsically impossible to carefully reason with. No matter how extreme we make the Pythagoreans' eccentricities, as long as they continue to do something math-ish, it would remain possible for a Euclid or Yudkowsky to arise from the sea-foam and propose a regimentation of their intuitions, a more carefully formalized version of their concepts of 'number,' 'ratio,' 'proof,' etc.
I take it that Eliezer thinks we are very much in the position today of inhabiting a global, heavily schismatized network of Pythagorean Cults of Morality. Those cults are irrational, and their favored concepts would need to be made more precise and careful before the questions they ask could be assigned determinate answers (even in principle). But the subject matter those cults are talking about -- how to cultivate human well-being, how to distribute resources equitably, how to balance preferences in a way most people would prefer, etc. -- is not intrinsically irrational or mystical or ineffable. The categories in question are tracking real property clusters, though perhaps not yet with complete applicability-to-any-old-case; no matter how much of a moral anti-realist you are, for instance, you can't reasonably hold that 'fairness' doesn't have its own set of satisfaction conditions that fail to coincide with other moral (or physical, mathematical, etc.) concepts.
Another way of motivating the idea that morality is 'logical': Decision theory is 'logical', and morality is a special sort of decision theory. If we can carefully regiment the satisfaction conditions for an individual's preferences, then we can regiment the satisfaction conditions for the preferences of people generally; and we can isolate the preferences that people consider moral vs. amoral; and if we can do all that, what skeptical challenge could block an algorithm that recognizably maps what we call 'fair' and 'unfair' and 'moral' and 'immoral,' that couldn't equally well block an algorithm that recognizably maps what we call 'preferred' and 'distasteful' and 'delicious'...? How carelessly do people have to reason with x such that we can conclude that it's impossible to reason carefully with x?
I think I've been careful not to claim that morality is impossible to carefully reason with, but just that we don't know how to carefully reason with it yet and given our current state of knowledge, it may turn out to be impossible to carefully reason with.
... (read more)