Re. your last remark, wouldn't a distinction between premise-circularity and rule-circularity do the trick?

I gestured at one possible answer to that question. A situation has a moral dimension if it engages moral emotions - which can presumably be listed.

@ Ian C. Couldn't Subhan claim that as a restatement of his own position? His notion of wanting clearly encompasses more than mere whims. Perhaps he would say that a certain subset of desires, objectively grounded in the constitution of the mind, count as moral impulses.

Actually, is Subhan meant to be male? Apologies if not.

Hear hear to Dynamically Linked's last paragraph.

There's at least one other intuition about the nature of morality to distinguish from the as-preference and as-given ideas. It's the view that there are only moral emotions - guilt, anger and so on - plus the situations that cause those emotions in different people. That's it. Morality on this view might profitably be compared with something like humour. Certain things cause amusement in certain people, and it's an objective fact that they do. At the same time, if two people fail to find the same thing funny, there wouldn't normally be any question of one of them failing to perceive some public feature of the world. And like the moral emotions, amusement is sui generis - it isn't reducible to preference, though it may often coincide with it. The idea of being either a realist or a reductionist about humour seems, I think, absurd. Why shouldn't the same go for morality?

I suppose that's just to second Paul Gowder's point that the political problem is insurmountable. But I imagine few things would resolve a political problem faster then the backing of an all-powerful supermind.

@Paul: You seem to suggest that we all take the same things to be reasons, perhaps even the same reasons. Is this warranted?

Eliezer: Are you looking for a new definition of "fairness" which would reconcile the partisans of existing definitions? Or are you just pointing out that this is a sort of damned-if-you-do, damned if-you-don't problem, and that any rule for establishing fairness will piss somebody or other off? If the latter, from the point of view of your larger project, why not just insert a dummy answer for this question - pick any definition that grabs you - and see how it fits with the rest of what you need to work out. Or work through several different obviously computable answers.

As fair as it goes, it seems plausible-ish that fairness has to do with equality of *something* - resources, or opportunity, or utility, or whatever - but I doubt whether there's any general agreement over what should be equalised, and I don't see the value of descending to a meta level of discussion to sort the question out. Meta-discussions would have to be answerable to fairness anyway, if they were to be fair, and that looks circular. So why not cut the knot and pick whatever answer is nearest to hand?

Hang on. @ Caledonian and Psy-Kosh: Surely mathematical language is just language that refers to mathematical objects - numbers and suchlike. Precise, unambiguous language doesn't count as mathematics unless it meets this condition.

You could call it "Overcoming Fun".

Eliezer's polemical tone is one of the great strengths of his pedagogical approach, IMO.

What, you mean you start finding it everywhere? If only.