I share your skepticism about Luke's statement (but I've been waiting to criticize until he finishes his sequence to see if he addresses the problems later).
My goal here is to get people to feel the intuition that "I ought to do X" means something, and that thing is not "I think I ought to do X" or "I would think that I ought to do X if I were smarter and some other stuff".
To help pump that intuition, consider this analogy:
"X is true" (where X is a mathematical statement) means something, and that thing is not "I think X is true" or "I would think that X is true if I were smarter and some other stuff".
On the other hand, I think it's also possible that "I ought to do X" doesn't really mean anything. See my What does a calculator mean by "2"?. (ETA: To clarify, I mean some usages of "ought" may not really mean anything. There are some usages that clearly do, for example "If you want to accomplish X, then you ought to do Y" can in principle be straightforwardly reduced to a mathematical statement about decision theory, assuming that our current strong intuition that there is such a thing as "the right decision theory" is correct.)
I'll first try to restate your position in order to check my understanding. Let me know if I don't do it justice.
People use "should" in several different ways. Most of these ways can be "reducible to physics", or in other words can be restated as talking about how our universe is, without losing any of the intended meaning. Some of these ways can't be so reduced (they are talking about the world of "is not") but those usages are simply meaningless and can be safely ignored.
I agree that many usages of "should" can be reduced to physics. (Or perhaps instead to mathematics.) But there may be other usages that can't be so reduced, and which are not clearly safe to ignore. Originally I was planning to wait for you to list the usages of "should" that can be reduced, and then show that there are other usages that are not obviously talking about "the world of is" but are not clearly meaningless either. (Of course I hope that your reductions do cover all of the important/interesting usages, but I'm not expecting that to be the case.)
Since you ask for my criticism now, I'll just give an example that seems to be one of the hardest to...
People who do feel that intuition run into trouble. This is because "I ought to do X' does not refer to anything that exists. How can you make a statement that doesn't refer to anything that exists?
It refers to my preferences which are physically encoded in my brain. It feels like it doesn't refer to anything that exists because I don't have complete introspective access to the mechanisms by which my brain decides that it wants something.
On top of that, ought refers to lots of different things, and as far as I can tell, most ought statements are su...
Will and I just spoke on the phone, so here's another way to present our discussion:
Imagine a species of artificial agents. These agents have a list of belief statements that relate physical phenomena to normative properties (let's call them 'moral primitives'):
These agents also have a list of belief statements about physical phenomena in general:
How can you make a statement that doesn't refer to anything that exists? I've done it, and my reasoning process is still intact, and nothing has blown up. Everything seems to be fine. No one has explained to me what isn't fine about this. Since it's intuitive, why would you not want to do it that way?
Clearly, you can make statements about things that don't exist. People do it all the time, and I don't object to it. I enjoy works of fiction, too. But if the aim of our dialogue is true claims about reality, then you've got to talk about things that exist ...
I made a more topical comment to Wei_Dai's reply to this thread, but I felt it was worth adding that if anyone is interested in a work of fiction that touches on this subject, the novel The Broom of the System, by David Foster Wallace, is worth a look.
Can you explain what implications (if any) this "naive" metaethics has on the problem how to build an FAI?
Let me have another go at this, since I've now rewritten the is-ought section of 'Pluralistic Moral Reductionism' (PMR).
This time around, I was more clear that of course it's true that, as you say:
we can make statements about what should be done and what should not be done that cannot be reduced, by definition, to statements about the physical world
We can make reducible 'ought' statements. We can make irreducible 'ought' statements. We can exhibit non-cognitive (non-asserting) verbal behaviors employing the sound 'ought'.
For the purposes of the PMR pos...
Suppose an AI wants to find out what Bob means when he says "water".
This thought experiment can be sharpened by asking what Bob means by "1027". By using Bob as an intermediary, we inevitably lose some precision.
People who do feel that intuition run into trouble. This is because "I ought to do X' does not refer to anything that exists. How can you make a statement that doesn't refer to anything that exists? I've done it, and my reasoning process is still intact, and nothing has blown up. Everything seems to be fine. No one has explained to me what isn't fine about this.
Ok, I'll bite. Why does "I ought to X" have to refer to any thing?
When I see atucker's comment, for instance;
...It refers to my preferences which are physically encoded in my brain.
"A Flatland Argument" includes a multitude of problems. It contains a few strawmen, which then turn into non sequiturs when you say that they demonstrate that we often say irreducible things. At least that's what I think you mean by the trivial statement "Not every set of claims is reducible to every other set of claims."
And yet your examples are answerable or full of holes! Why not mention the holes rather than blaming the reductionism (or, in the case of some strawmen, total lack thereof), or mention the reduction and forget the wh...
It seems clear to me that I can multiply about what I care about, so I don't know quite what you want to say.
Since it's intuitive, why would you not want to do it that way?
What seems wrong with the obvious answer?
Do you think a FOOM'd self-modifying AI that cares about humanity's CEV would likely do what you consider 'right'? Why or why not? (If you object to the question, please address that issue separately.)
The verb believe goes without saying when making claims about the world. To assert that 'the sign is red' is true would not make sense if I did not believe it, by definition. I would either be lying or unaware of my own mental state. To me, your question borders more on opinions and their consequences.
Quoting from there: "But your beliefs are not about you; beliefs are about the world. Your beliefs should be your best available estimate of the way things are; anything else is a lie."
What I'm trying to say is that the statement (Murder is wrong) implies the further slight linguistic variant (I believe murder is wrong) (modulo the possibility that someone is lying or mentally ill, etc.) The question then is whether (I believe murder is wrong) -> (murder is wrong). Ultimately, from the perspective of the person making these claims, the answer is 'yes'. It makes no sense for me to feel that my preferences are not universally and unequivocally true.
I don't find this at odds with a situation where a notorious murderer who is caught, say Hannibal Lecter, can simultaneously choose his actions and say "murder is wrong". Maybe the person is mentally insane. But even if they aren't, they could simply choose a preference ordering such that the local wrongness of failing to gratify their desire to murder is worse than the local wrongness of murder itself in their society. Thus, they can see that to people who don't have the same preference for murdering someone for self-gratification, the computation of beliefs works out that (murder is wrong) is generally true, but not true when you substitute their local situations into their personal formula for computing the belief. In this case it just becomes an argument over words because the murderer is tacitly substituting his personal local definitions for things when making choices, but then using more general definitions when making statements of beliefs. In essence, the murderer believes it is not wrong for him to murder and get the gratification, but that murder, as society defines it and views it, is "wrong" where "wrong" is a society-level description, not the murderer's personal description. I put a little more about the "words" problem below.
The apparent difference between this way of thinking and the way we all experience our thinking is that, among our assertions is the meta-assertion that (over-asserting beliefs is bad) -> (I believe over-asserting beliefs is bad) or something similar to this. All specific beliefs, including such meta-beliefs, are intertwined. You can't have independent beliefs about whether murder is right that don't depend on your beliefs about whether beliefs should be acted upon like they are cold hard facts.
But at the root, all beliefs are statements about physics. Mapping a complicated human belief down to the level of making statistical pattern recognition claims about amplitude distributions is really hard and inaccessible to us. Further, evolutionarily, we can't afford to burn computation time exploring a fully determined picture of our beliefs. After some amount of computation time, we have to make our chess moves or else the clock runs out and we lose.
It only feels like saying (I believe murder is wrong) fails to imply the claim (murder is wrong). Prefacing a claim with "I believe" is a human-level way or trying to mitigate the harshness of the claim. It could be a statement that tries to roughly quantify how much evidence I can attest to for the claim which the belief describes. It certainly sounds more assured to say (murder is wrong) than to say (I believe murder is wrong), but this is a phantom distinction.
The other thing, which I think you are trying to take special pains to avoid, is that you can very easily run into a battle of words here. If someone says, "I believe murder is wrong" and what they really mean is something like "I believe that it does an intolerable amount of social disservice in the modern society that I live in for anyone to act as if murdering is acceptable, and thus to always make sure to punish murderers," basically, if someone translates "murder" into "the local definition of murder in the world that I frequently experience" and they translate "wrong" into "the local definition of wrong (e.g. punishable in court proceedings or something)", then they are no longer talking about the cognitive concept of murder. An alien race might not define murder the same or "wrong" the same.
If someone uses 'believe' to distinguish between making a claim about the most generalized form of murder they can think of, applicable to the widest array of potential sentient beings, or something like that, then the two statements are different, but only artificially.
If I say "I believe murder is wrong" and I really mean "I believe (my local definition of murder) is (my local definition of wrong)" then this implies the statement (The concept described by my local definition of murder is locally wrong), with no "quantifier" of belief required.
In the end, all statements can be reduced this way. If a statement has "I believe" as a "quantifier", then either it is only an artificial facet of language that restricts the definitions of words in the claim to some (usually local) subset on which the full, unprefaced claim can be made... or else if local definitions of words aren't being implicated, then the "I believe" prefix literally contains no additional information about the state of your mind than the raw assertion would yield.
This is why rhetoric professors go nuts when students write argumentative papers and drop "I think that" or "I believe that" all over the place. Assertions are assertions. It's a social custom that you can allude to the fact that you might not have 100% confidence in your assertion by prefacing it with "I believe". It's also a social custom that you can allude to respect for other beliefs or participation in a negotiation process by prefacing claims with "I believe", but in the strictest sense of what information you're conveying to third parties (separate from any social custom dressings), the "I believe" preface adds no information content.
The difference is here
Alice: "I bet you $500 that the sign is red" Bob: "OK" later, they find out it's blue Bob: "Pay up!"
Alice: "I bet you $500 that I believe the sign is red" Bob: "OK" later, they find out it's blue Alice: "But I thought it was red! Pay up!"
That's the difference between "X" and "I believe X". We say them in the same situation, but they mean different things.
...But even if they aren't, they could simply choose a preference ordering such that the local wrongnes
I aim to make several arguments in the post that we can make statements about what should be done and what should not be done that cannot be reduced, by definition, to statements about the physical world.
A Naive Argument
Lukeprog says this in one of his posts:
I would like to question that statement. I would guess that lukeprog's chief subject of interest is figuring out what to do with the options presented to him. His interest is, therefore, in figuring out what he ought to do.
Consider the reasoning process that takes him from observations about the world to actions. He sees something, and then thinks, and then thinks some more, and then decides. Moreover, he can, if he chooses, express every step of this reasoning process in words. Does he really lose interest at the last step?
My goal here is to get people to feel the intuition that "I ought to do X" means something, and that thing is not "I think I ought to do X" or "I would think that I ought to do X if I were smarter and some other stuff".
(If you don't, I'm not sure what to do.)
People who do feel that intuition run into trouble. This is because "I ought to do X' does not refer to anything that exists. How can you make a statement that doesn't refer to anything that exists?
I've done it, and my reasoning process is still intact, and nothing has blown up. Everything seems to be fine. No one has explained to me what isn't fine about this.
Since it's intuitive, why would you not want to do it that way?
(You can argue that certain words, for certain people, do not refer to what one ought to do. But it's a different matter to suggest that no word refers to what one ought to do beyond facts about what is.)
A Flatland Argument
"I'm not interested in words, I'm interested in things. Words are just sequences of sounds or images. There's no way a sequence of arbitrary symbols could imply another sequence, or inform a decision."
"I understand how logical definitions work. I can see how, from a small set of axioms, you can derive a large number of interesting facts. But I'm not interested in words without definitions. What does "That thing, over there?" mean? Taboo finger-pointing."
"You can make statements about observations, that much is obvious. You can even talk about patterns in observations, like "the sun rises in the morning". But I don't understand your claim that there's no chocolate cake at the center of the sun. Is it about something you can see? If not, I'm not interested."
"Claims about the past make perfect sense, but I don't understand what you mean when you say something is going to happen. Sure, I see that chair, and I remember seeing the chair in the past, but what do you mean that the chair will still be there tomorrow? Taboo "will"."
Not every set of claims is reducible to every other set of claims. There is nothing special about the set "claims about the state of the world, including one's place in it and ability to affect it." If you add, however, ought-claims, then you will get a very special set - the set of all information you need to make correct decisions.
I can't see a reason to make claims that aren't reducible, by definition, to that.
The Bootstrapping Trick
Suppose an AI wants to find out what Bob means when he says "water'. AI could ask him if various items were and were not water. But Bob might get temporarily confused in any number of ways - he could mix up his words, he could hallucinate, or anything else. So the AI decides instead to wait. The AI will give Bob time, and everything else he needs, to make the decision. In this way, by giving Bob all the abilities he needs to replicate his abstract concept of a process that decides if something is or is not "water", the AI can duplicate this process.
The following statement is true:
But this is certainly not the definition of water! Imagine if Bob used this criterion to evaluate what was and was not water. He would suffer from an infinite regress. The definition of water is something else. The statement "This is water" reduces to a set of facts about this, not a set of facts about this and Bob's head.
The extension to morality should be obvious.
What one is forced to do by this argument, if one wants to speak only in physical statements, is to say that "should" has a really, really long definition that incorporates all components of human value. When a simple word has a really, really long definition, we should worry that something is up.
Well, why does it have a long definition? It has a long definition because that's what we believe is important. To say that people who use (in this sense) "should" to mean different things just disagree about definitions is to paper over and cover up the fact that they disagree about what's important.
What do I care about?
In this essay I talk about what I believe about rather than what I care about. What I care about seems like an entirely emotional question to me. I cannot Shut Up And Multiply about what I care about. If I do, in fact, Shut Up and Multiply, then it is because I believe that doing so is right. Suppose I believe that my future emotions will follow multiplication. I would have to, then, believe that I am going to self-modify into someone who multiplies. I would only do this because of a belief that doing so is right.
Belief and logical reasoning are an important part of how people on lesswrong think about morality, and I don't see how to incorporate them into a metaethics based not on beliefs, but on caring.