What do I mean by "morality isn't logical"? I mean in the same sense that mathematics is logical but literary criticism isn't: the "reasoning" we use to think about morality doesn't resemble logical reasoning. All systems of logic, that I'm aware of, have a concept of proof and a method of verifying with high degree of certainty whether an argument constitutes a proof. As long as the logic is consistent (and we have good reason to think that many of them are), once we verify a proof we can accept its conclusion without worrying that there may be another proof that makes the opposite conclusion. With morality though, we have no such method, and people all the time make moral arguments that can be reversed or called into question by other moral arguments. (Edit: For an example of this, see these posts.)
Without being a system of logic, moral philosophical reasoning likely (or at least plausibly) doesn't have any of the nice properties that a well-constructed system of logic would have, for example, consistency, validity, soundness, or even the more basic property that considering arguments in a different order, or in a different mood, won't cause a person to accept an entirely different set of conclusions. For all we know, somebody trying to reason about a moral concept like "fairness" may just be taking a random walk as they move from one conclusion to another based on moral arguments they encounter or think up.
In a recent post, Eliezer said "morality is logic", by which he seems to mean... well, I'm still not exactly sure what, but one interpretation is that a person's cognition about morality can be described as an algorithm, and that algorithm can be studied using logical reasoning. (Which of course is true, but in that sense both math and literary criticism as well as every other subject of human study would be logic.) In any case, I don't think Eliezer is explicitly claiming that an algorithm-for-thinking-about-morality constitutes an algorithm-for-doing-logic, but I worry that the characterization of "morality is logic" may cause some connotations of "logic" to be inappropriately sneaked into "morality". For example Eliezer seems to (at least at one point) assume that considering moral arguments in a different order won't cause a human to accept an entirely different set of conclusions, and maybe this is why. To fight this potential sneaking of connotations, I suggest that when you see the phrase "morality is logic", remind yourself that morality isn't logical.
The practice of moral philosophy doesn't much resemble the practice of mathematics. Mainly because in moral philosophy we don't know exactly what we're talking about when we talk about morality. In mathematics, particularly since the 20th century, we can eventually precisely specify what we mean by a mathematical object, in terms of sets.
"Morality is logic" means that when we talk about morality we are talking about a mathematical object. The fact that the only place in our mind the reference to this object is stored is our intuition is what makes moral philosophy so difficult and non-logicy. In practice you can't write down a complete syntactic description of morality, so in general¹ neither can you write syntactic proofs of theorems about morality. This is not to say that such descriptions or proofs do not exist!
In practice moral philosophy proceeds by a kind of probabilistic reasoning, which might be analogized to the thinking that leads one to conjecture that P≠NP, except with even less rigor. I'd expect that things like the order of moral arguments mattering come down to framing effects and other biases which are always involved regardless of the subject, but don't show up in mathematics so much because proofs leave little wiggle room.
¹ Of course, you may be able to write proofs that only use simple properties that you can be fairly sure hold of morality without knowing its full description, but such properties are usually either quite boring or not widely agreed upon or don't lead to interesting proofs due to being too specific. eg. "It's wrong to kill someone without their permission when there's nothing to be gained by it."
How does one go about defining this mathematical object, in principle? Suppose you were a superintelligence and could surmount any kind of technical difficulty, and you wanted to define a human's morality precisely as a mathematical object, how would you do it?