jsalvati - "I think the difference is that in a world where one of them is miscalculating, that person can be shown that they are miscalculating and will then calculate correctly."

This still won't do, due to path-dependence and such. Suppose Bob could be corrected in any number of ways, and each will cause him to adopt a different conclusion -- and one that he will then persist in holding no matter what other arguments you give him. Which conclusion is the true value for our original morality_Bob? There can presumably be no fact of the matter, on Eliezer's account. And if this sort of underdetermination is very common (which I imagine it is), then there's probably no facts at all about what any of our "moralities" are. There may always be some schedule of information that would bring us to make radically different moral judgments.

Also worrying is the implication that it's impossible to be stubbornly wrong. Once you become impervious to argument in your adoption of inconsistent moral beliefs, well, those contradictions are now apparently part of your true morality, which you're computing just fine.(?)

Psy-Kosh - "Well... the computation your brain is, under the hood, performing when you're trying to figure out things about "what should I do?""

That just pushes my question back a step. Don't the physical facts underdetermine what computation ('abstracted idealized dynamic') my brain might be interpreted as performing? It all depends how you abstract and idealize it, after all. Unless, that is, we think there's some brute (irreducible) facts about which are the right idealizations...

HA - "what resources do you recommend I look into to find people taking a more rigorous approach to understanding the phenomenon of human morality"

If you're interested in the empirical phenomenon, I'm the wrong person to ask. (Maybe start with the SEP on moral psychology?) But on a philosophical level I'd recommend Peter Railton for a sophisticated naturalistic metaethic (that I respect a lot while not entirely agreeing with). He has a recent bloggingheads diavlog, but you can't go past his classic article 'Moral Realism' [here if you have jstor access].

Eliezer - that's all well and good, but what in the world do you think determines which computation or 'abstract idealized dynamic' a mortal human is actually referring to? Won't this be radically underdetermined?

You suggest that "Bob and Sally could be talking about different things when they talk about Enamuh". What's the difference between a world where they're talking about different things vs. a world where they are talking about the same thing but one of them is 'miscalculating'? What facts (about their dispositions and such) would determine which of the two explanations holds, on your view?

Carl - "If you're going to define 'fully reasonable' to mean sharing your moral axioms, so that a superintelligent pencil maximizer with superhuman understanding of human ethics and philosophy is not a 'reasonable agent,' doesn't this just shift the problem a level? Your morality_objectivenorms is only common to all agents with full reasonableness_RichardChappell, and you don't seem to have any compelling reason for the latter (somewhat gerrymandered) account of reasonableness save that it's yours/your culture's/your species.'"

I don't mean to define 'fully reasonable' at all (though it is meant to be minimally ad hoc or gerrymandered). I take this normative notion as a conceptual primitive, and then hypothesize that it entails a certain set of moral norms. They're probably not even my norms (in any way Eliezer could accommodate), since I'm presumably not fully reasonable myself. But they're what I'm trying to aim for, even if I don't always grasp them correctly.

This may sound mysterious and troublingly ungrounded to you. Yet you use terms like 'superintelligent' and 'superhuman understanding', which are no less normative than my 'reasonable'. I think that reasonableness is a component of intelligence and (certainly) understanding, so I don't see how these terms could properly apply to a pencil maximizer. Maybe you simply mean that it is a pencil maximizer that is instrumentally rational and perfectly proficient at Bayesian updating. But that's not to say it's intelligent. It might, for example, be a counterinductivist (didn't someone mention anti-Occamists up-thread?), with completely wacky priors. I take it as a datum that this is simply unreasonable -- there are other norms, besides conditionalization and instrumental rationality, which govern 'intelligent' or good thinking.

So I say there are brute, unanalysable facts about what's reasonable. The buck's gotta stop somewhere. I don't see that any alternative theory does better than this one.

"Perhaps Richard means that we could suppose that abortion is indeed prohibited by morality_Bob..."

That's right. (I didn't mean to suggest that there's never any disputing what someone's moral commitments are; just that this wasn't supposed to be in dispute in the particular case I was imagining.) I take it that Sally and Bob could disagree even so, and not merely be talking past each other, even if one or both of them was impervious to rational argument. It is at least a significant cost of your theory that it denies this datum. (It doesn't have to be 'irrefutable' to nonetheless be a hefty bullet to bite!)

I like your account of everyday moral disagreement, but would take it a step further: it is no mere accident that your Sally and Bob expect to be able to persuade the other. Rather, it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents.

It's worth emphasizing, though, that humans are not fully reasonable. Some are even irrevocably unreasonable, incapable of rationally updating (some of) their beliefs. So while, I claim, we all aspire to the morality_Objective norms, I doubt there's any empirically specifiable procedure that could ensure our explicit affirmation of those norms (let alone their unfolded implications). Bob may stubbornly insist that abortion is wrong, and this may conflict with other claims he makes, but there's simply no way (short of brain surgery) to shake him from his illogic. What then? I say he's mistaken, even though he can't be brought to recognize this himself. It's not clear to me whether you can say this, since I'm not sure exactly what your 'extrapolation' procedure for defining morality_Bob is. But if it's based on any simple empirical facts about what Bob would believe if we told him various facts and arguments, then it doesn't look like you'll be able to correct for the moral errors that result from sheer irrationality, or imperviousness to argument.

Could you say a little more about exactly which empirical facts serve to define morality_Bob?

Larry, not that the particular example is essential to my point, but you're clearly not familiar with the strongest pro-life arguments.

"My notion of goodness may be slightly different to yours - how can we have a sensible conversation where you insist on using the word "morality" to refer to morality_Eliezer2008?"

This is an important objection, which I think establishes the inadequacy of Eliezer's analysis. It's a datum (which any adequate metaethical theory must account for) that there can be substantive moral disagreement. When Bob says "Abortion is wrong", and Sally says, "No it isn't", they are disagreeing with each other.

I don't see how Eliezer can accommodate this. On his account, what Bob asserted is true iff abortion is prohibited by the morality_Bob norms. How can Sally disagree? There's no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob. On the other hand, it would be changing the subject for Sally to say "Abortion is right" in her own vernacular, where this merely means that abortion is permitted by the morality_Sally norms. (Bob wasn't talking about morality_Sally, so their two claims are - on Eliezer's account - quite compatible.)

Since there is moral disagreement, whatever Eliezer purports to be analysing here, it is not morality.

[For more detail, see 'Is Normativity Just Semantics?]

I've an old post - 'Verification and Base Facts' - which shows how a non-verificationist can still capture much of what was most compelling in verificationism.

I second Unknown. It's worth noting that trolls like Caledonian also deter other (more reasonable) voices from joining the conversation, so it's not at all clear that his contributions promote dissent on net. (And I think it is clear that they don't promote reasonable dissent.)