Right, so my point is that if your theory (that moral reasoning is probabilistic reasoning about some mathematical object) is to be correct, we need a definition of morality as a mathematical object which isn't "what X says after considering all possible moral arguments". So what could it be then? What definition Y can we give, such that it makes sense to say "when we reason about morality, we are really doing probabilistic reasoning about the mathematical object Y"?
Unfortunately I doubt I can give you a short direct definition of morality. However if such a mathematical object exists, "what X says after considering all possible moral arguments" should be enough to pin it down (disregarding the caveats to do with our subject going insane, etc).
Secondly, until we have a candidate definition Y at hand, we can't show that moral reasoning really does correspond to probabilistic logical reasoning about Y. (And we'd also have to first understand what "probabilistic logical reasoning" is.) So, at this point, how can we be confident that moral reasoning does correspond to probabilistic logical reasoning about anything mathematical, and isn't just some sort of random walk or some sort of reasoning that's different from probabilistic logical reasoning?
Well, I think it safe to assume I mean something by moral talk, otherwise I wouldn't care so much about whether things are right or wrong. I must be talking about something, because that something is wired into my decision system. And I presume this something is mathematical, because (assuming I mean something by "P is good") you can take the set of all good things, and this set is the same in all counterfactuals. Roughly speaking.
It is, of course, possible that moral reasoning isn't actually any kind of valid reasoning, but does amount to a "random walk" of some kind, where considering an argument permanently changes your intuition in some nondeterministic way so that after hearing the argument you're not even talking about the same thing you were before hearing it. Which is worrying.
Also it's possible that moral talk in particular is mostly signalling intended to disguise our true values which are very similar but more selfish. But that doesn't make a lot of difference since you can still cash out your values as a mathematical object of some sort.
It is, of course, possible that moral reasoning isn't actually any kind of valid reasoning, but does amount to a "random walk" of some kind, where considering an argument permanently changes your intuition in some nondeterministic way so that after hearing the argument you're not even talking about the same thing you were before hearing it. Which is worrying.
Yes, exactly. This seems to me pretty likely to be the case for humans. Even if it's actually not the case, nobody has done the work to rule it out yet (has anyone even written a post maki...
What do I mean by "morality isn't logical"? I mean in the same sense that mathematics is logical but literary criticism isn't: the "reasoning" we use to think about morality doesn't resemble logical reasoning. All systems of logic, that I'm aware of, have a concept of proof and a method of verifying with high degree of certainty whether an argument constitutes a proof. As long as the logic is consistent (and we have good reason to think that many of them are), once we verify a proof we can accept its conclusion without worrying that there may be another proof that makes the opposite conclusion. With morality though, we have no such method, and people all the time make moral arguments that can be reversed or called into question by other moral arguments. (Edit: For an example of this, see these posts.)
Without being a system of logic, moral philosophical reasoning likely (or at least plausibly) doesn't have any of the nice properties that a well-constructed system of logic would have, for example, consistency, validity, soundness, or even the more basic property that considering arguments in a different order, or in a different mood, won't cause a person to accept an entirely different set of conclusions. For all we know, somebody trying to reason about a moral concept like "fairness" may just be taking a random walk as they move from one conclusion to another based on moral arguments they encounter or think up.
In a recent post, Eliezer said "morality is logic", by which he seems to mean... well, I'm still not exactly sure what, but one interpretation is that a person's cognition about morality can be described as an algorithm, and that algorithm can be studied using logical reasoning. (Which of course is true, but in that sense both math and literary criticism as well as every other subject of human study would be logic.) In any case, I don't think Eliezer is explicitly claiming that an algorithm-for-thinking-about-morality constitutes an algorithm-for-doing-logic, but I worry that the characterization of "morality is logic" may cause some connotations of "logic" to be inappropriately sneaked into "morality". For example Eliezer seems to (at least at one point) assume that considering moral arguments in a different order won't cause a human to accept an entirely different set of conclusions, and maybe this is why. To fight this potential sneaking of connotations, I suggest that when you see the phrase "morality is logic", remind yourself that morality isn't logical.