I now get what you meant by keeping the different 'levels' or layers separate – that notion, or record-keeping, of what's happening, in your proof, in PA versus PA+1. I loved my math courses precisely because of the computational logic.

Your meta-ethics: We're not right, nor will any of our descendants [?] ever be right. But we know some things that ARE right, just as a matter of inevitable ev-bio competence. And we've figured out a few ways to learn new things that are right. We've explored a bit of right, and even a bit of right+1. But we don't want to go near self-right – if we ASSUME we're accurate in our current understanding of right or right+1, we can (falsely) justify anything.

"I can only plead that when I look over my flawed mind and see a core of useful reasoning, that I am really right, even though a completely broken mind might mistakenly perceive a core of useful truth."

"humans have received a moral gift, which Pebblesorters lack, in that we started out interested in things like happiness instead of just prime pebble heaps. Now this is not actually a case of someone reaching in from outside with a gift-wrapped box... it is only when you look out from within the perspective of morality, that it seems like a great blessing that there are humans around"

Quick question: do you intend the latter deflationary remarks to apply to your 'epistemic gift' too? That is, would you emphasize that your methods of reasoning are merely considered to be a gift from within your own perspective, and there's not any further sense to the notion of 'good priors' or a 'broken mind' or 'useful reasoning' beyond the brute fact that you happen to use these words to refer to these particular epistemic norms? Or do you think there's an important difference between the kind of (moral vs. epistemic) 'mistakes' made respectively by the Pebblesorters and the anti-Inductors?

With these distinctions in mind, I hope my intent will be clearer, when I say that although I am human and have a human-ish moral framework, I do not think that the fact of acting in a human-ish way licenses anything.

hah. I was wondering what this Lob stuff had to do with morality.

good job on an excellent post.

"But of course I made a very deliberate decision not to speak of "h-right". That sounds like there is a general license to be human."

Okay, this is a good point, and a good post.

I still think, however, you're left with the empirical question of how strongly psychological unity applies to moral dynamics: to what extent (if any) different people are just different optimization processes with nothing to argue about.

Eliezer: Good post, as always, I'll repeat that I think you're closer to me in moral philosophy than anyone else I've talked to, with the probable exception of Richard Rorty, from whom I got many of my current views. (You might want to read Contingency, Irony, Solidarity; it's short, and it talks about a lot of the stuff you deal with here). That said, I disagree with you in two places. Reading your stuff and the other comments has helped me refine what I think; I'll try to state it here as clearly as possible.

1) I think that, as most people use the words, you're a moral relativist. I understand why you think you're not. But the way most people use the word 'morality,' it would only apply to an argument that would persuade the ideal philosopher of perfect emptiness. You don't believe any such arguments exist; neither do I. Thus neither of us think that morality as it's commonly understood is a real phenomenon. Think of the priest in War of the Worlds who tried to talk to the aliens, explaining that since we're both rational beings/children of God, we can persuade them not to kill us because it's wrong. You say (as I understand you) that they would agree that it's wrong, and just not care, because wrong isn't necessarily something they care about. I have no problem with any claim you've made (well, that I've made on your behalf) here; but at this point the way you're using the word 'moral' isn't a way most people would use it. So you should use some other term altogether.

2) I like to maintain a clearer focus on the fact that, if you care about what's right, I care about what's right_1, which is very similar to but not the same as what's right. Mainly because it helps me to remember there are some things I'm just not going to convince other people of (e.g. I don't think I could convince the Pope that God doesn't exist. There's no fact pattern that's wholly inconsistent with the property god_exists, and the Pope has that buried deep enough in its priors that I don't think it's possible to root it out). But (as of reading your comment on yesterday's post) I don't think we disagree on the substance, just on the emphasis.

Thanks for an engaging series of posts; as I said, I think you're the closest or second-closest I've ever come across to someone sharing my meta-ethics.

Ah, but who is deciding whether I possess good priors? I am! By reflecting on them! ... I can only plead that ... I have been blessed with an epistemic gift.

It is only a coincidence that in these posts you have only described yourself better than hypothetical creatures? Do actual creatures tend to do better than the examples you've given?

If I understand this correctly, it would make the point clearer to distinguish between implementation of optimization process and a passive concept of right. Humans are intelligent agents that, among other things, use a concept of right in their cognitive algorithm. It is hard to separate this concept from the process of applying it in the specific way it's done in humans, but this concept is not about the way its implementation is interpreted. You can't appeal to the implementation of humans when talking about this concept, the question that only includes the physical structure of humans (plus adaptive environment) is not sufficient to lead to an answer that describes the concept of right, because you also need to know what to look for in this implementation, how to interpret information about this concept present in the interpretation, how to locate interfaces that allow to read out the information about it. Talking about right requires interpretation from the inside, and communicating the concept requires building the interpreter based on this inside view.

The full implementation of a human doesn't contain enough information to ask a question about what is right, without interpretation by a human. Given interpretation by a human, implementation only helps in arriving to a more precise form of the question, in moving towards the answer, by assembling a more coherent form of it against factual knowledge about the process by which implementation functions.

Yvain, good anecdote about Darrow. I hate spinach too.

Say half the population takes a pill that makes them really, truly believe that murder is right. The way I understand Eliezer's assertion that his morals aren't relative, he'd say 'no, murder is still wrong', and would probably assert the same even if 100% of the population took the pills. The pill-takers would assert, and absolutely believe, that they were right. Not p-right, but right. I'd love to hear proof that the pill-takers are wrong, and that everyone else is right. Not p-right, but right.

Murder is always wrong to the extent that those abstract complex calculations will still come out the same. Is that the extent of the argument? If so, what does it even mean to say 'even if everyone believes murder is right, it's still wrong'? Where does that data exist? Smacks of dualism to me.

Yvain: "that's why I think Eliezer's system is relative. I admit it's not directly relative, in that Eliezer isn't directly picking "Don't murder" out of belief-space every time he wonders about murder, based only on human opinion. But if I understand correctly, he's referring the question to another layer, and then basing that layer on human opinion.

An umpire whose procedure for making tough calls is "Do whatever benefits the Yankees" isn't very fair. A second umpire whose procedure is "Always follow the rules in Rulebook X" and writes in Rulebook X "Do whatever benefits the Yankees" may be following a rulebook, but he is still just as far from objectivity as the last guy was.

I think the second umpire's call is "correct" relative to Rulebook X, but I don't think the call is absolutely correct."

• I'd agree with this analysis. Well put.

@Ben Jones: "Say half the population takes a pill that makes them really, truly believe that murder is right. The way I understand Eliezer's assertion that his morals aren't relative, he'd say 'no, murder is still wrong', and would probably assert the same even if 100% of the population took the pills. The pill-takers would assert, and absolutely believe, that they were right. Not p-right, but right. I'd love to hear proof that the pill-takers are wrong, and that everyone else is right. Not p-right, but right."

This comment underlines the fact that Eliezer's use of language is not the standard one. According to Eliezer's usage, it is not possible to opine that murder is right, in the same way that it is not possible to opine that 22 is prime. Eliezer has defined "right" to be a specific constant set of terminal values. Unfortunately, he hasn't ever specified that constant fully, just like if I said "x is the real number 8.1939... etc". [What comes after the dots?]

Asking whether the pill-takers are "right" is, in Eliezer's terminology, like asking if x = 0, where x is simply defined to be the real number 8.1939.. etc: it's false in an obvious way. Nothing is right, except what Eliezer has in mind when he writes lists of values followed by "...", just like no real number is equal to 8.1939... etc, except, um, 8.1939... etc.

This is, of course, not the way most people use the word "right". They use it as the name of a variable to hold the answer to the question "how shall I live?".

Roko: It certainly is possible to opine that 22 is prime. Watch this:

22 is prime!

See, I did it. If you claim murder is right, then you aren't talking about something other than right, you are just making false statements about right.

Asking whether the pill-takers are "right" is, in Eliezer's terminology, like asking if x = 0, where x is simply defined to be the real number 8.1939.. etc: it's false in an obvious way. Nothing is right, except what Eliezer has in mind when he writes lists of values followed by "..."

That number could be given the name 'right', but right then has all other meaning stripped away from it. More importantly, the choice of number seems to be arbitrary.

Constants like e and pi are definable in terms of certain important properties that we discovered through analysis of problems that are often important in real-life situations. What makes right a special number in the same way that e and pi are special?

In fact, I have a good analogy for the naming problem that Eliezer has created: suppose we are physicists and we are trying to work out how fast light travels through free space. We take measurements of distance and time between various locations, and decide to denote the average of these measurements by the symbol "c".

Eliezer decides to start using the symbol "c" to denote the real number 3*10^8.

Ben Jones then asks whether c might actually equal 2*10^8.

Others asked whether c might equal 299,792 ,458. No! Of course it doesn't, says Eliezer: c is defined to be 300,000,000.

"c" --- "right" "300,000,000" --- "the output of the CEV algorithm"/"the average human moral viewpoint"

Some of these (e.g. Roko's) concerns might be clarified in terms of the distinctions between sense, reference, and reference-fixing descriptions. I take it Eliezer wants to use 'right' as a rigid designator to denote some particular set of terminal values, but others have pointed out that this reference fact is fixed by means of a seemingly 'relative' procedure (namely, whatever terminal values he himself happens to hold, on some appropriate [if somewhat mysterious] idealization). There is also some concern that this doesn't match the plain meaning or sense of the term 'right', as everyone else understands it.

Roko:

Eliezer decides to start using the symbol "c" to denote the real number 3*10^8.

No, he has continuously refused to spell out an explicit description of morality, because it admits no concise description. When Eliezer writes a list of values ending with "etcetera" he's saying (in your analogy) "c is 3*10^8, up to one significant digit".

Richard:

but others have pointed out that this reference fact is fixed by means of a seemingly 'relative' procedure

I think you are mixing meta-levels here. The seemingly relative procedure is used to describe morality in blog posts, not to chose what morality is in the first place.

Larry, no, the mix-up is yours. I didn't say anything about morality, I was talking about the word 'right', and the meta-semantic question how it is that this word refers to rightness (some particular combination of terminal values) rather than, say, p-rightness.

Richard: It seems to me that asking how is it that the word 'right' came to refer to rightness is like asking why 'green' means green, instead of meaning zebra.

The fact is that there is some concept that we've been calling "right", and though we don't exactly know what we mean by it, we're pretty certain it means something, and in some cases we know it when we see it.

It strikes me as unfair to accuse Eliezer of having his own private meaning of "right" that isn't in accordance with the common one, because hasn't endorsed a criterion or decision procedure for 'right', he hasn't tried to define it, he hasn't made clearly-wrong claims about it like "murder is right", he really hasn't said much of anything about the object-level practical meaning of 'right'. He has mostly just discussed certain meta-level features of the concept, such as the fact that isn't all-possible-minds-universal, and the idea that one who explicitly thinks "If i think X is right, then X is right" can think that anything is right.

the idea that one who explicitly thinks "If i think X is right, then X is right" can think that anything is right.

It strikes me as unfair to accuse Eliezer of having his own private meaning of "right" that isn't in accordance with the common one

That is incompatible with the common meaning of rightness.

Caledonian: That's relativism, right there - the idea that rightness is not only socially determined, but individually socially determined.

What!? That's just not what I said at all.

"asking how is it that the word 'right' came to refer to rightness is like asking why 'green' means green"

Yeah, that's not exactly what I meant. As I see it there are two stages: there's the question how the symbols 'right' (or 'green') get attached to the concept that they do, and then there's the more interesting question of how this broad sense of the term determines -- in combination with the actual facts -- what the term actually refers to, i.e. what property the concept denotes. So I should have asked how it is that our sense of the concept 'right', as we hold it in our minds, determines what external property is ultimately denoted by the term. (Compare how the concept 'water' ultimately denotes the property of being H2O.) It is this step of Eliezer's account, I think, which looks to some to be suspiciously relativistic, and in conflict with the sense of the term as they understand it. Maybe he's picking out the right property (hard to tell when he's said so little about it, as you say). But the meta-properties, the concept, the procedure by which what we have in mind picks out a particular thing in the word, that just seems all wrong.

We have already escaped moral-relativity; as evidence, I submit our current discussion. No self-optimizing decision procedure (e.g. for right, p-right, ...) can begin with nothing. Each one is a gift existence grants itself.

This talk about metaethics is trying to justify building castles in the clouds by declaring the foundation to be supported by the roof. It doesn't deal with the fundamental problem at all - it makes it worse.

Caledonian, I don't want to speak for Eliezer. But my contention, at least, is that the fundamental problem is insoluble. I claim, not that this particular castle has a solid foundation, but that there exist no solid foundations, and that anywhere you think you've found solid earth there's actually a cloud somewhere beneath it. The fact that you're reacting so strongly makes me think you're interpreting Eliezer as saying what I believe. Similarly,

Why should we care about a moral code that Eliezer has arbitrarily chosen to call right? What relevance does this have to anything?

There's no particular reason we should care about a moral code Eliezer has chosen. You should care about the moral code you have arbitrarily chosen. I claim, and I think Eliezer would too, that there will be a certain amount of overlap because you're both human (just as you both buy into Occam because you're both human). But we couldn't give, say, a pebblesorter any reason to care about Eliezer's moral code.

Larry D'ana: Is anyone who does not believes in universally compelling arguments a relativist?

Is anyone who does not believe that morality is ontologically primitive a relativist?

Yeah, pretty much.

If there are no universally compelling arguments, then there's no universally compelling moral code. Which means that whatever code compels you has to compel relative to who you are; thus it's a relativist position.

Eliezer tries to get around this by saying that he has this code he can state (to some low degree of precision), and everyone can objectively agree on whether or not some action comports with this code. Or at least that perfect Bayesian superintelligences could all agree. (I'm not entirely sold on that, but we'll stipulate). I claim, though, that this isn't the way most people (including most of us) use the words 'morality' and 'right'; I think that if you want your usage to comport with everyone else's, you would have to say that the pebblesorters have 'a' moral code, and that this moral code is "Stack pebbles in heaps whose sizes are prime numbers."

In other words, in general usage a moral code is a system of rules that compels an agent to action (and has a couple other properties I haven't figured out how to describe without self-reference). A moral absolutist claims that there exists such a system of rules that is rightly binding and compelling to all X, where X is usually some set like "all human beings" or "all self-aware agents." (Read e.g. Kant who claimed that the characteristic of a moral rule is that it is categorically binding on all rational minds). But Eliezer and I claim that there are no universally compelling arguments of any sort. Thus in particular there are no universally compelling injunctions to act, and thus no absolute moral code. Instead, the injunction to act that a particular agent finds compelling varies with the identity of the agent; thus 'morality' is relative to the agent. And thus I'm a moral relativist.

Now, it's possible that you could get away with restricting X to "human beings"; if you then claimed that humans had enough in common that the same moral code was compelling to all of them, you could plausibly reclaim moral objectivism. But I think that claim is clearly false; Eliezer seems to have rejected it (or at least refused to defend it) as well. So we don't get even that degree of objectivity; the details of each person's moral code depend on that person, and thus we have a relative standard. This is what has Caledonian's knickers in such a twist.

Kenny: exactly. That's why we're morally relative.

You should care about the moral code you have arbitrarily chosen.

If there are no universally compelling arguments, then there's no universally compelling moral code.

@J Thomas – We are born with some theorems of right (in analogy to PA). We are not blank slates. That is our escape hatch from the abyss of self-right (i.e. moral relativity). We have already been granted the gift of a (small) part of right. Again, it is not h-right, but right – just as it is not h-PA, but simply PA.

We are born with some theorems of right (in analogy to PA).

Kenny, I'd be fascinated to learn more about that. I didn't notice it in my children, but then I wouldn't necessarily notice.

When I was a small child people claimed that babies are born with only a fear of falling and a startle reflex for loud noises. I was pretty sure that was wrong, but it wasn't clear to me what we're born with. It takes time to learn to see. I remember when I invented the inverse square law for vision, and understood why things get smaller when they go farther away. It takes time to notice that parents have their own desires that need to be taken into account.

What is it that we're born with? Do you have a quick link maybe?

"You should care about the moral code you have arbitrarily chosen."

No, I shouldn't. Which seems to be the focal point of this endless 'debate'.

Well, you might choose to care about a moral code you have arbitrarily chosen. And it could be argued that if you don't care about it then you haven't "really" chosen it.

I agree with you that there needn't be any platonic absolute morality that says you ought choose a moral code arbitrarily and care about it, or that if you do happen to choose a moral code arbitrarily that you should then care about it.

Well, you might choose to care about a moral code you have arbitrarily chosen.

1) I don't think it's possible to choose in such a way - what I care about is not directly controllable by my conscious awareness. It is sometimes possible for me to set up circumstances so that my emotional responses are slowly directed in one way instead of another, but it's slow and chancey.

2) I assert that caring about arbitrarily-chosen stances is wrong.

I wouldn't describe my position as 'Platonic', but there is a limited degree of similarity. If there are no objective moral realities which we can attempt, however crudely and imperfectly, to model in our understanding, I assert that caring about moral stances is incorrect. That isn't what caring is for - self-referntially being concerned about our positions and inclinations is pointless if it makes no difference what position we choose.

Caledonian, it's possible to care deeply about choices that were made in a seemingly-arbitrary way. For example, a college graduate who takes a job in one of eight cities where he got job offers, might within the year care deeply about that city's baseball team. But if he had taken a different job it would be a completely different baseball team.

You might care about the result of arbitrary choices. I don't say you necessarily will.

It sounds like you're saying it's wrong to care about morals unless they're somehow provably correct? I'm not sure I get your objection. I want to point out that usually when we have a war, most of the people in each country choose sides based on which country they are in. Less than 50% of americans chose to oppose the current iraq fiasco before it happened. Imagine that russia had invaded iraq with the same pretexts we used, all of which would have worked as well for russia as well as they did for us. Russians had more reason than us to fear iraqi nukes, they didn't want iraq supporting chechen terrorists, they thought Saddam was bad man, etc. imagine the hell we would have raised about it.... But I contend that well over a hundred million americans supported the war for no better reason that they were born in america and so they supported invasions by the US military.

Whether or not there's some higher or plausible morality that says we should not choose our morals at random, still the fact is that most of us do choose our morals at random.

But this does not constitute a disagreement between them and humans about what is right, any more than humans, in scattering a heap of 3 pebbles, are disagreeing with the Pebblesorters about which numbers are prime!

That is an excellent compression of the arguments against the idea that humans and Pebblesorters are actually disagreeing about anything. (I don't think I found that unclear at any point, but this sentence has a distinctly and pleasantly intuition-pumpy feel.)

Does this sound a little less indefensible, if I mention that PA trusts only proofs from the PA axioms, not proofs from every possible set of axioms?

This makes me wonder if something interesting might be said about a system that does trust proofs from every possible set of axioms. Or a system that consists of every possible axiom.

It would be inconsistent, obviously, but what else?

This is now in the running for my favorite posts of the sequences.