= 27d567bc2d48d9f40e9ccc20ea370b15)
A Defense of Naive Metaethics
I aim to make several arguments in the post that we can make statements about what should be done and what should not be done that cannot be reduced, by definition, to statements about the physical world.
A Naive Argument
Lukeprog says this in one of his posts:
If someone makes a claim of the 'ought' type, either they are talking about the world of is, or they are talking about the world of is not. If they are talking about the world of is not, then I quickly lose interest because the world of is not isn't my subject of interest.
I would like to question that statement. I would guess that lukeprog's chief subject of interest is figuring out what to do with the options presented to him. His interest is, therefore, in figuring out what he ought to do.
Consider the reasoning process that takes him from observations about the world to actions. He sees something, and then thinks, and then thinks some more, and then decides. Moreover, he can, if he chooses, express every step of this reasoning process in words. Does he really lose interest at the last step?
My goal here is to get people to feel the intuition that "I ought to do X" means something, and that thing is not "I think I ought to do X" or "I would think that I ought to do X if I were smarter and some other stuff".
(If you don't, I'm not sure what to do.)
People who do feel that intuition run into trouble. This is because "I ought to do X' does not refer to anything that exists. How can you make a statement that doesn't refer to anything that exists?
I've done it, and my reasoning process is still intact, and nothing has blown up. Everything seems to be fine. No one has explained to me what isn't fine about this.
Since it's intuitive, why would you not want to do it that way?
(You can argue that certain words, for certain people, do not refer to what one ought to do. But it's a different matter to suggest that no word refers to what one ought to do beyond facts about what is.)
A Flatland Argument
"I'm not interested in words, I'm interested in things. Words are just sequences of sounds or images. There's no way a sequence of arbitrary symbols could imply another sequence, or inform a decision."
"I understand how logical definitions work. I can see how, from a small set of axioms, you can derive a large number of interesting facts. But I'm not interested in words without definitions. What does "That thing, over there?" mean? Taboo finger-pointing."
"You can make statements about observations, that much is obvious. You can even talk about patterns in observations, like "the sun rises in the morning". But I don't understand your claim that there's no chocolate cake at the center of the sun. Is it about something you can see? If not, I'm not interested."
"Claims about the past make perfect sense, but I don't understand what you mean when you say something is going to happen. Sure, I see that chair, and I remember seeing the chair in the past, but what do you mean that the chair will still be there tomorrow? Taboo "will"."
Not every set of claims is reducible to every other set of claims. There is nothing special about the set "claims about the state of the world, including one's place in it and ability to affect it." If you add, however, ought-claims, then you will get a very special set - the set of all information you need to make correct decisions.
I can't see a reason to make claims that aren't reducible, by definition, to that.
The Bootstrapping Trick
Suppose an AI wants to find out what Bob means when he says "water'. AI could ask him if various items were and were not water. But Bob might get temporarily confused in any number of ways - he could mix up his words, he could hallucinate, or anything else. So the AI decides instead to wait. The AI will give Bob time, and everything else he needs, to make the decision. In this way, by giving Bob all the abilities he needs to replicate his abstract concept of a process that decides if something is or is not "water", the AI can duplicate this process.
The following statement is true:
A substance is water (in Bob's language) if and only if Bob, given all the time, intelligence, and other resources he wants, decides that it is water.
But this is certainly not the definition of water! Imagine if Bob used this criterion to evaluate what was and was not water. He would suffer from an infinite regress. The definition of water is something else. The statement "This is water" reduces to a set of facts about this, not a set of facts about this and Bob's head.
The extension to morality should be obvious.
What one is forced to do by this argument, if one wants to speak only in physical statements, is to say that "should" has a really, really long definition that incorporates all components of human value. When a simple word has a really, really long definition, we should worry that something is up.
Well, why does it have a long definition? It has a long definition because that's what we believe is important. To say that people who use (in this sense) "should" to mean different things just disagree about definitions is to paper over and cover up the fact that they disagree about what's important.
What do I care about?
In this essay I talk about what I believe about rather than what I care about. What I care about seems like an entirely emotional question to me. I cannot Shut Up And Multiply about what I care about. If I do, in fact, Shut Up and Multiply, then it is because I believe that doing so is right. Suppose I believe that my future emotions will follow multiplication. I would have to, then, believe that I am going to self-modify into someone who multiplies. I would only do this because of a belief that doing so is right.
Belief and logical reasoning are an important part of how people on lesswrong think about morality, and I don't see how to incorporate them into a metaethics based not on beliefs, but on caring.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Comments (294)
Will and I just spoke on the phone, so here's another way to present our discussion:
Imagine a species of artificial agents. These agents have a list of belief statements that relate physical phenomena to normative properties (let's call them 'moral primitives'):
These agents also have a list of belief statements about physical phenomena in general:
These agents also have an 'ought' function that includes a series of logical statements that relate normative concepts to each other, such as:
Finally, these robots have actuators that are activated by a series of rules like:
Some of these rules might include utility functions that encode ordinal or cardinal value for varying combinations of normative properties.
These agents can't see their own source code. The combination of the moral primitives and the ought function and the non-ought belief statements and a set of rules about behavior produces their action and their verbal statements about what ought to be done.
From their behavior and verbal ought statements these robots can infer to some degree how their ought function works, but they can't fully describe their ought function because they haven't run enough tests or the ought function is just too complicated or the problem is made worse because they also can't see their moral primitives.
The ought function doesn't reduce to physics because it's a set of purely logical statements. The 'meaning' of ought in this sense is determined by the role that the ought function plays in producing intentional behavior by the robots.
Of course, the robots could speak in ought language in stipulated ways, such that 'ought' means 'that which produces pleasure in human brains' or something like that, and this could be a useful way to communicate efficiently, but it wouldn't capture what the ought function is doing or how it is contributing to the production of behavior by these agents.
What Will is saying is that it's convenient to use 'ought' language to refer to this ought function only, and not also to a combination of the ought function and statements about physics, as happens when we stipulatively use 'ought' to talk about 'that which produces well-being in conscious creatures' (for example).
I'm saying that's fine, but it can also be convenient (and intuitive) for people to use 'ought' language in ways that reduce to logical-physical statements, and not only in ways that express a logical function that contains only transformations between normative properties. So we don't have substantive disagreement on this point; we merely have different intuitions about the pragmatic value of particular uses for 'ought' language.
We also drew up a simplified model of the production of human action in which there is a cognitive module that processes the 'ought' function (made of purely logical statements like in the robots' ought function), a cognitive module that processes habits, a cognitive module that processes reflexes, and so on. Each of these produces an output, and another module runs arg(max) on these action options to determine which actions 'wins' and actually occurs.
Of course, the human 'ought' function is probably spread across multiple modules, as is the 'habit' function.
Will likes to think of the 'meaning' of 'ought' as being captured by the algorithm of this 'ought' function in the human brain. This ought function doesn't contain physical beliefs, but rather processes primitive normative/moral beliefs (from outside the ought function) and outputs particular normative/moral judgments, which contribute to the production of human behavior (including spoken moral judgments). In this sense, 'ought' in Will's sense of the term doesn't reduce to physical facts, but to a logical function.
I'm fine with Will using 'ought' in that sense if he wants. I'll try to be clear how I am using the term when I use it.
Will also thinks that the 'ought' function (in his sense) inside human brains is probably very similar between humans - ones that aren't brain damaged or neurologically deranged. I don't know how probable this is because cognitive neuroscience hasn't progressed that far. But if the 'ought' function is the same in all healthy humans, then there needn't be a separate 'meaning' of ought (in Will's sense) for each speaker, but instead there could be a shared 'meaning' of ought (in Will's sense) that is captured by the algorithms of the 'ought' cognitive module that is shared by healthy human brains.
Will, did I say all of that correctly?
That doesn't seem right. Compare (note that I don't necessarily endorse the rest of this paper) :
If you examine just one particular sense of the word "ought", even if you make clear which sense, but without systematically enumerating all of the meanings of the word, how can you know that the concept you end up studying is the one that is actually important, or one that other people are most interested in?
I suspect there are many senses of a word like 'ought' that are important. As 'pluralistic moral reductionism' states, I'm happy to use and examine multiple important meanings of a word.
Let me expand my comment a bit, because it didn't quite capture what I wanted to say.
If Will is anything like a typical human, then by "ought" he often means something other than, or more than, the sense referred to by "that sense", and it doesn't make sense to say that perhaps he wants to use "ought" in that sense.
When you say "I'm fine with ..." are you playing the role of the Austere Metaethicist who says "Tell me what you mean by 'right', and I will tell you what is the right thing to do."? But I think Austere Metaethics is not a tenable metaethical position, because when you ask a person to tell you what they mean by "right", they will almost certainly fail to give you a correct answer, simply because nobody really understands (much less can articulate) what they mean by "right". So what is the point of that?
Or perhaps what you meant to say instead was "I'm fine with Will studying 'ought' in that sense if he wants"? In that case see my grandparent comment (but consider it directed mostly towards Will instead of you).
I don't love all your terminology, but obviously my preferred terminology's ability to communicate my ideas on this matter has been shown to be poor.
I would emphasize less relationships between similar moral beliefs:
and more the assembly-line process converting general to specific
I'm pretty sure the first statement here only makes sense as a consequence of the second:
This doesn't make sense to me. Does 28 reduce to physics in this sense? How is this "ought" thing distinguished from all the other factors (moral errors, say) that contribute to behavior (that is, how is its role located)?
First, I would say that reducibility is a property of statements. In the sense I use it:
The statement "14+14=28" is reducible to aether.
The statement "I have 28 apples" is reducible to phyisics.
The statement "There are 28 fundamental rules that one must obey to lead a just life" is reducible to ethics.
Moral statements are irreducible to physics in the sense that "P is red" is irreducible to physics - for any particular physical "P", it is reducible. The logical properties of P-statements, like "P is red or P is not red" are given as a set of purely logical statements - that's their analogue of the ought-function. If P-statements had some useful role in producing behavior, they would have a corresponding meaning.
Random, probably unnecessary math:
A reducible-class is a subalgebra of the Boolean algebra of statements, closed under logical equivalence. The statements reducible to aether are those in the reducible-class generated by True and False. The statements reducible to physics are those in the reducible-class generated by "The world is in exactly state X". The statements reducible to morality are those in the reducible-class generated by "Exactly set-of-actions Y are forbidden and set-of-actions Z are obligatory".
Let me have another go at this, since I've now rewritten the is-ought section of 'Pluralistic Moral Reductionism' (PMR).
This time around, I was more clear that of course it's true that, as you say:
We can make reducible 'ought' statements. We can make irreducible 'ought' statements. We can exhibit non-cognitive (non-asserting) verbal behaviors employing the sound 'ought'.
For the purposes of the PMR post, I'm interested to investigate frameworks that allow us to determine whether a certain subset of 'ought' statements are true or false.
Thus, in the context of PMR, non-asserting verbal behaviors employing 'ought' sounds are simply a different subject matter. They do not belong to the subset of 'ought' statements I am investigating for that particular post. (I discussed this in the cognitivism vs. non-cognitivism section.) At the same time, clearly such statements can be useful. For example, you can use them to affect others' behavior. You can affect others' behavior and attitudes with non-asserting verbal behaviors, such as when you say "female circumcisions" with a gasp and a frown.
Also in the context of PMR, some 'ought' statements can quickly be tossed in the 'false' bin, because the speaker uses 'ought' language assertively to refer to things that clearly don't exist, like divine commands.
In the context of PMR, other 'ought' statements can be tossed into the 'false' bin if you are a physicalist, because the speaker uses 'ought' language assertively to refer to things that don't fit within a physicalist ontology, like non-natural moral properties. If you're not a physicalist, then our debate about such 'ought' statements can shift to a debate about physicalism vs. non-physicalism.
Will, you seem to be saying that 'ought' has only one meaning, or one definition. You also seem to be saying that this one meaning of 'ought' (or 'should') is captured by what we 'actually want' in the CEV sense. Is that right so far?
If so, I'm still not clear on your arguments for this conclusion. Your writing here has a very high ratio of unstated premises to stated premises. My own writing does that all the time for communication efficiency, in the hopes that my unstated premises are shared or else successfully inferred. But many times I find that the unstated premises didn't make it into the other's mind, and thus my enthymeme is unclear to it. And when I care enough about my argument being clear to certain people, I take the time to draw my unstated premises into the light and state them explicitly.
Since I'm having trouble guessing at the unstated premises in your arguments for singularism about the meaning of 'should' or 'ought', I'll request that you state them explicitly. Would you please state your most central argument for this singularism, without unstated premises?
What are your alternatives (at this level of detail)? If I could be using two different definitions, ought1 and ought2, then I expect there are distinguishing arguments that form a decision problem about which of the two I should've been using, which in turn determines which of these definitions is the one.
Well there are cases when I should be using two different words.
For instance, if morality is only one component of the correct decision procedure, then MoralOught and CorrectOught are two different things.
But you're not talking about those types of cases, right?
Don't understand what you said. Probably not.
Well, suppose that sometimes, depending on context cues, I use "ought" to mean "paperclip-maximizing", "prime-pile-maximizing", and "actually-ought".
There's nothing wrong about the first two definitions, they're totally reasonable definitions a word might have, they just shouldn't be confused with the third definition, which specifies correct actions.
Well, I am saying that there is a meaning of "ought" that is hugely different in meaning from the other senses.
PMR identifies a sort of cluster of different meanings of the word "ought". I am saying, hey, over here, there's this one, singular meaning.
This meaning is special because it has a sense but no referent. It doesn't refer to any property of the physical world, or obviously, of any property of any non-physical world. It just means.
[Not CEV, will explain later with time.]
Okay. I look forward to it.
So in this perspective what I "want" is really a red herring. I want to do lots of things that I oughtn't do.
What matters is my beliefs about what is right and wrong.
Now, by necessity, I believe that my EV is the best possible approximation of what is right. Because, If I knew of a better approximation, I would incorporate it into my beliefs, and if I didn't know of it, my volition must not have been extrapolated far enough.
But this is not a definition of what is right. To do so would be circular.
If I believe that my EV is very close to humanity's CEV, then I believe that humanity's CEV is almost the best approximation as to what is right. I do, so I do.
So, to start reasoning, I need assumptions. My assumptions would look like:
or
or something else, just as the assumptions I use to generate physical beliefs would consist of my intuitions about the proper techniques for induction (Bayesianism, Occam's Razor, and so on.)
There doesn't have to be any Book O' Right sitting around for me to engage in this reasoning, I can just, you know, do it.
(It is very ironic that I first developed this edifice because I was bothered by unstated moral assumptions.)
I'm confused by your way of presenting your arguments and conclusion. On my end this comment looks like a list of unconnected thoughts, with no segues between them. Does somebody else think they know what Will is saying, such that they can explain it to me?
I drew some boundaries between largely-though-not-totally unconnected thoughts.
Does everything within those boundaries look connected to you?
I think Vladimir Nesov agrees with me on this.
Thanks, but it's still not clear to me. Nesov, do you want to take a shot and arguing for Will's position, especially if you agree with it?
Clearly, you can make statements about things that don't exist. People do it all the time, and I don't object to it. I enjoy works of fiction, too. But if the aim of our dialogue is true claims about reality, then you've got to talk about things that exist - whether the subject matter is 'oughts' or not.
I don't see why this needs to be the case. I can stipulate short meanings of 'should' as I use the term. People do this all the time (implicitly, at least) when using hypothetical imperatives.
Also, in general I find myself confused by your way of talking about these things. It's not a language I'm familiar with, so I suspect I'm still not fully understanding you. I'm not sure which of our anticipations differ because of the disagreement you're trying to express.
But the aim of our dialogue isn't really true claims. It's useful claims - claims that one can incorporate into one's decision-making process. Claims about Darth Vader, you can't, but claims about ought, you can.
What about that other word (that is also spelled "should") that you don't have to stipulate the meaning of because people already know what it means?
What about the regular kind of imperatives?
If I define "fa" to mean "any object which more than 75% of the claims in this long book of no previous importance accurately describes", I have done something very strange, even if I letter say "If 'fa' means 'red', that's fa."
I don't understand what you mean, here. I'm not sure what you mean by 'true' or 'useful', I guess. I'm talking about true claims in this sense.
Which one is that, and what does everybody already know it to mean?
I mean what you mean by "true", or maybe something very similar.
By "useful" I mean "those claims that could help someone come to a decision about their actions"
It's what people say when they say "should" but don't precede it with "if". Some people on lesswrong think it means:
[you should do X] = [X maximizes this complicated function that can be computed from my brain state]
Some think it means:
[you should do X] = [X maximizes whatever complicated function is computed from my brain state]
and I think:
[you should do X] = [the statement that, if believed, would cause one to do X]
or rather [you should do X] = [the statement that, if believed, would cause one to do X if one were an ideal and completely non akrasic agent]
Correct.
You can find that there is a bug in your brain that causes you to react to a certain belief, but you'd fix it if you notice it's there, since you don't think that belief should cause that action.
I could say
[the statement that, if believed by a rational agent, would cause it to do X]
but that's circular.
But one of the points I've been trying to make is that it's okay for the definition of something to be, in some sense, circular. As long as you can describe the code for a rational agent that manipulates that kind of statement.
Some things you can't define exactly, only refer to them with some measure of accuracy. Physical facts are like this. Morality is like this. Rational agents don't define morality, they respond to it, they are imperfect detectors of moral facts who would use their moral expertise to improve own ability to detect moral facts or build other tools capable of that. There is nothing circular here, just constant aspiration for referencing the unreachable ideal through changeable means.
But there aren't causal arrows pointing from morality to rational agents, are there? Just acausal/timeless arrows.
You do have to define "morality" as meaning "that thing that we're trying to refer to with some measure of accuracy", whereas "red" is not defined to refer to the same thing.
If you agree, I think we're on the same page.
I think the idea of acausal/logical control captures what causality was meant to capture in more detail, and is a proper generalization of it. So I'd say that there are indeed "causal" arrows from morality to decisions of agents, to the extent the idea of "causal" dependence is used correctly and not restricted to the way we define physical laws on a certain level of detail.
Why would I define it so? It's indeed what we are trying to refer to, but what it is exactly we cannot know.
Lost me here. We know enough about morality to say that it's not the same thing as "red", yes.
Sure.
Let me rephrase a bit.
"That thing, over there (which we're trying to refer to with some measure of accuracy), point point".
I'm defining it extensionally, except for the fact that it doesn't physically exist.
There has to be some kind of definition or else we wouldn't know what we were talking about, even if it's extensional and hard to put into words.
"red" and "right" have different extensional definitions.
All definitions should be circular. "The president is the Head of State" is a correct definition. "The president is Obama" is true, but not a definition.
Non-circular definitions can certainly be perfectly fine:
"A bachelor is an unmarried man.'
This style is used in math to define new concepts to simplify communication and thought.
"A bachelor is an unmarried man.'
If that is non circular, so is [the statement that, if believed by a rational agent, would cause it to do X]
I'm quite confused. By circular do you mean anaylitcal, or recursive? (example of the latter: a setis something that can contain elemetns or other sets)
I'm not sure what I mean.
The definition I am using is in the following category:
It may appear problematically self-referential, but it is in fact self-referential in a non-problematic manner.
Agreed?
Which would mean either that mathematical knowledge is false, or that there is a Platonic word of mathematical objects for it to correspond to.
OTOH, one could just adopt the Dogma of Empiricism that there is analytical truth which is neither 'about' physical realitty nor 'about' about any metaphysical one ( and that mathematical truth is anayltical). (and that mathematical truth is anayltical).
And if it is an analytical truth that, for instance, that you should do as you would be done by, then that is still applicable to real world situations by fulling "as you would be done by" for your own case.
I share your skepticism about Luke's statement (but I've been waiting to criticize until he finishes his sequence to see if he addresses the problems later).
To help pump that intuition, consider this analogy:
"X is true" (where X is a mathematical statement) means something, and that thing is not "I think X is true" or "I would think that X is true if I were smarter and some other stuff".
On the other hand, I think it's also possible that "I ought to do X" doesn't really mean anything. See my What does a calculator mean by "2"?. (ETA: To clarify, I mean some usages of "ought" may not really mean anything. There are some usages that clearly do, for example "If you want to accomplish X, then you ought to do Y" can in principle be straightforwardly reduced to a mathematical statement about decision theory, assuming that our current strong intuition that there is such a thing as "the right decision theory" is correct.)
Wei Dai,
I would prefer to hear the source of your skepticism now, if possible. I anticipate not actually disagreeing. I anticipate that we will argue it out and discover that we agree but that my way of expressing my position was not clear to you at first. And then I anticipate using this information to improve the clarity of my future posts.
I'll first try to restate your position in order to check my understanding. Let me know if I don't do it justice.
People use "should" in several different ways. Most of these ways can be "reducible to physics", or in other words can be restated as talking about how our universe is, without losing any of the intended meaning. Some of these ways can't be so reduced (they are talking about the world of "is not") but those usages are simply meaningless and can be safely ignored.
I agree that many usages of "should" can be reduced to physics. (Or perhaps instead to mathematics.) But there may be other usages that can't be so reduced, and which are not clearly safe to ignore. Originally I was planning to wait for you to list the usages of "should" that can be reduced, and then show that there are other usages that are not obviously talking about "the world of is" but are not clearly meaningless either. (Of course I hope that your reductions do cover all of the important/interesting usages, but I'm not expecting that to be the case.)
Since you ask for my criticism now, I'll just give an example that seems to be one of the hardest to reduce: "Should I consider the lives of random strangers to have (terminal) value?"
(Eliezer's proposal is that what I'm really asking when I ask that question is "Does my CEV think the lives of random strangers should have (terminal) value?" I've given various arguments why I find this solution unsatisfactory. One that is currently fresh on my mind is that "coherent extrapolation" is merely a practical way to find the answer to any given question, but should not be used as the definition of what the question means. For example I could use a variant of CEV (call it Coherent Extrapolated Pi Estimation) to answer "What is the trillionth digit of pi?" but that doesn't imply that by "the trillionth digit of pi" I actually mean "the output of CEPE".)
I'm not planning to list all the reductions of normative language. There are too many. People use normative language in too many ways.
Also, I should clarify that when I talk about reducing ought statements into physical statements, I'm including logic. On my view, logic is just a feature of the language we use to talk about physical facts. (More on that if needed.)
I'm not sure I would say "most."
What do you mean by "safe to ignore"?
If you're talking about something that doesn't reduce (even theoretically) into physics and/or a logical-mathematical function, then what are you talking about? Fiction? Magic? Those are fine things to talk about, as long as we understand we're talking about fiction or magic.
What about this is hard to reduce? We can ask for what you mean by 'should' in this question, and reduce it if possible. Perhaps what you have in mind isn't reducible (divine commands), but then your question is without an answer.
Or perhaps you're asking the question in the sense of "Please fix my broken question for me. I don't know what I mean by 'should'. Would you please do a stack trace on the cognitive algorithms that generated that question, fix my question, and then answer it for me?" And in that case we're doing empathic metaethics.
I'm still confused as to what your objection is. Will you clarify?
That's making a pre-existing assumption that everyone speaks in physics language. It's circular.
Speaking in physic language about something that isn't in the actual physics is fiction. I'm not sure what magic is.
What is physics language? Physics language consists of statements that you can cash out, along with a physical world, to get "true" or "false"
What is moral language? Moral language consists of statements that you can cash out, along with a preference order on the set of physical worlds, to get "true" or "false".
ETA: IF you don't accept this, the first step is accepting that the statement "Flibber fladoo." does not refer to anything in physics, and is not a fiction.
No, of course lots of people use 'ought' terms and other terms without any reduction to physics in mind. All I'm saying is that if I'm right about reductionism, those uses of ought language will fail to refer.
Sure, that's one way to use moral language. And your preference order is computed by physics.
That's the way I'm talking about, so you should be able to ignore the other ways in your discussion with me.
You are proposing a function MyOrder from {states of the world} to {preference orders}
This gives you a natural function from {statements in moral language} to {statements in physical language}
but this is not a reduction, it's not what those statements mean, because it's not what they're defined to mean.
I think I must be using the term 'reduction' in a broader sense than you are. By reduction I just mean the translation of (in this case) normative language to natural language - cashing things out in terms of lower-level natural statements.
But you can't reduce an arbitrary statement. You can only do so when you have a definition that allows you to reduce it. There are several potential functions from {statements in moral language} to {statements in physical language}. You are proposing that for each meaningful use of moral language, one such function must be correct by definition.
I am saying, no, you can just make statements in moral language which do not correspond to any statements in physical language.
Not what I meant to propose. I don't agree with that.
Of course you can. People do it all the time. But if you're a physicalist (by which I mean to include Tegmarkian radical platonists), then those statements fail to successfully refer. That's all I'm saying.
You said that you're not interested in an "ought" sentence if it reduces to talking about the world of is not. I was trying to make the same point by "safe to ignore".
I don't know, but I don't think it's a good idea to assume that only things that are reducible to physics and/or math are worth talking about. I mean it's a good working assumption to guide your search for possible meanings of "should", but why declare that you're not "interested" in anything else? Couldn't you make that decision on a case by case basis, just in case there is a meaning of "should" that talks about something else besides physics and/or math and its interestingness will be apparent once you see it?
Maybe I should have waited until you finish your sequence after all, because I don't know what "doing empathic metaethics" actually entails at this point. How are you proposing to "fix my question"? It's not as if there is a design spec buried somewhere in my brain, and you can check my actual code against the design spec to see where the bug is... Do you want to pick up this conversation after you explain it in more detail?
Maybe this is because I'm fairly confident of physicalism? Of course I'll change my mind if presented with enough evidence, but I'm not anticipating such a surprise.
'Interest' wasn't the best word for me to use. I'll have to fix that. All I was trying to say is that if somebody uses 'ought' to refer to something that isn't physical or logical, then this punts the discussion back to a debate over physicalism, which isn't the topic of my already-too-long 'Pluralistic Moral Reductionism' post.
Surely, many people use 'ought' to refer to things non-reducible to physics or logic, and they may even be interesting (as in fiction), but in the search for true statements that use 'ought' language they are not 'interesting', unless physicalism is false (which is a different discussion, then).
Does that make sense? I'll explain empathic metaethics in more detail later, but I hope we can get some clarity on this part right now.
Dude, you really need to start distinguishing between reducible-in-principle and usefully-reducible and doesn't need-reducing.
First I would call myself a radical platonist instead of a physicalist. (If all universes that exist mathematically also exist physically, perhaps it could be said that there is no difference between platonism and physicalism, but I think most people who call themselves physicalists would deny that premise.) So I think it's likely that everything "interesting" can be reduced to math, but given the history of philosophy I don't think I should be very confident in that. See my recent How To Be More Confident... That You're Wrong.
Right, I'm pretty partial to Tegmark, too. So what I call physicalism is compatible with Tegmark. But could you perhaps give an example of what it would mean to reduce normative language to a logical-mathematical function - even a silly one?
(It's late and I'm thinking up this example on the spot, so let me know if it doesn't make sense.)
Suppose I'm in a restaurant and I say to my dinner companion Bob, "I'm too tired to think tonight. You know me pretty well. What do you think I should order?" From the answer I get, I can infer (when I'm not so tired) a set of joint constraints on what Bob believes to be my preferences, what decision theory he applied on my behalf, and the outcome of his (possibly subconscious) computation. If there is little uncertainty about my preferences and the decision theory involved, then the information conveyed by "you should order X" in this context just reduces to a mathematical statement about (for example) what the arg max of a set of weighted averages is.
(I notice an interesting subtlety here. Even though what I infer from "you should order X" is (1) "according to Bob's computation, the arg max of ... is X", what Bob means by "you should order X" must be (2) "the arg max of ... is X", because if he means (1), then "you should order X" would be true even if Bob made an error in his computation.)
Yeah, that's definitely compatible with what I'm talking about when I talk about reducing normative language to natural language (that is, to math/logic + physics).
Do you think any disagreements or confusion remains in this thread?
Do you accept the conclusion I draw from my version of this argument?
You'd need the FAI able to change its mind as well, which requires that you retain this option in its epistemology. To attack the communication issue from a different angle, could you give examples of the kinds of facts you deny? (Don't say "god" or "magic", give a concrete example.)
Yes, we need the FAI to be able to change its mind about physicalism.
I don't think I've ever been clear about what people mean to assert when they talk about things that don't reduce to physics/math.
Rather, people describe something non-natural or supernatural and I think, "Yeah, that just sounds confused." Specific examples of things I deny because of my physicalism are Moore's non-natural goods and Chalmers' conception of consciousness.
SInce you can't actually reduce[*] 99.99% of your vocabulary, you're either so confused you couldn't possibly think or communicate...or you're only confused about the nature of confusion.
[*] Try reducing "shopping" to quarks, electrons and photons.You can't do it, and if you could, it would tell you nothing useful. Yet there is nothing that is not made of quarks,electrons and photons involved.
Not much better than "magic", doesn't help.
Is this because you're not familiar with Moore on non-natural goods and Chalmers on consciousness, or because you agree with me that those ideas are just confused?
Logic can be used to talk about non-physical facts. Do you allow referring to logic even where the logic is talking about non-physical facts, or do you only allow referring to the logic that is talking about physical facts? Or maybe you taboo intended interpretation, however non-physical, but still allow the symbolic game itself to be morally relevant?
Alas, I think this is getting us into the problem of universals. :)
With you, too, Vladimir, I suspect our anticipations do not differ, but our language for talking about these subtle things is slightly different, and thus it takes a bit of work for us to understand each other.
By "logic referring to non-physical facts", do you have in mind something like "20+7=27"?
"3^^^^3 > 3^^^3", properties of higher cardinals, hyperreal numbers, facts about a GoL world, about universes with various oracles we don't have.
Things for which you can't build a trivial analogy out of physical objects, like a pile of 27 rocks (which are not themselves simple, but this is not easy to appreciate in the context of this comparison).
Certainly, one could reduce normative language into purely logical-mathematical facts, if that was how one was using normative language. But I haven't heard of people doing this. Have you? Would a reduction of 'ought' into purely mathematical statements ever connect up again to physics in a possible world? If so, could you give an example - even a silly one?
Since it's hard to convey tone through text, let me explicitly state that my tone is a genuinely curious and collaboratively truth-seeking one. I suspect you've done more and better thinking on metaethics than I have, so I'm trying to gain what contributions from you I can.
Why do you talk of "language" so much? Suppose we didn't have language (and there was only ever a single person), I don't think the problem changes.
Say, I would like to minimize ((X-2)*(X-2)+3)^^^3, where X is the number I'm going to observe on the screen. This is a pretty self-contained specification, and yet it refers to the world. The "logical" side of this can be regarded as a recipe, a symbolic representation of your goals. It also talks about a number that is too big to fit into the physical world.
Okay, sure. We agree about this, then.
This would require that we both have positions that accurately reflect reality, or are somehow synchronously deluded. This is a confusing territory, I know that I don't know enough to be anywhere confident in my position, and even that position is too vague to be worth systematically communicating, or to describe some important phenomena (I'm working on that). I appreciate the difficulty of communication, but I don't believe that we would magically meet at the end without having to change our ideas in nontrivial ways.
I just mean that our anticipations do not differ in a very local sense. As an example, imagine that we were using 'sound' in different ways like Albert and Barry. Surely Albert and Barry have different anticipations in many ways, but not with respect to the specific events closely related to the tree falling in a forest when nobody is around.
Or maybe things that just don't usefully reduce.
I'd be very gracious if you could take a look at my recent question and the comments. Your statement
is interesting to me. What is a counter-argument to the claim that the only way that one could claim that " "X is true" means something" is to unpack the statement "X is true" all the way down to amplitudes over configurations (perhaps in a subspace of configuration space that highly factorizes over 'statistically common arrangements of particles in human brains correlating to mathematical conclusions' or something.
Where do the intuition-sympathizers stand on the issue of logical names?
I don't think something like 'ought' can intuitively point to something that has ontological ramifications. If there is any "intuition" to it, why is it unsatisfactory to think it's merely an evolutionary effect?
From the original post above, I find a point of contention with
'I ought to do X' does correspond to something that exists... namely, some distribution over configurations of human minds. It's a proposition like any other, like 'that sign is red' for example. You can track down a fully empirical and quantifiable descriptor of 'I ought to do X' with some sufficiently accurate model and measuring devices with sufficient precision. States of knowledge about what one 'ought' to do are states of knowledge like any others. When tracking down the physics of 'Ought', it will be fleshed out with some nuanced, perhaps situationally specific, definition that relates it to other existing entities.
I guess more succinctly, there is no abstract concept of 'ought'. The label 'ought' just refers to an algorithm A, an outcome desired from that algorithm O, an input space of things the algorithm can operate on, X, an assessment of the probability that the outcome happens under the algorithm, P(A(X) = O). Up to the limit of sensory fidelity, this is all in principle experimentally detectable, no?
I don't believe in an ontology of morals, only an epistemology of them.
Do you think that "The sign is red" means something different from "I believe the sign is red"? (In the technical sense of believe, not the pop sense.)
Do you think that "Murder is wrong" means something different from "I believe that murder is wrong."?
The verb believe goes without saying when making claims about the world. To assert that 'the sign is red' is true would not make sense if I did not believe it, by definition. I would either be lying or unaware of my own mental state. To me, your question borders more on opinions and their consequences.
Quoting from there: "But your beliefs are not about you; beliefs are about the world. Your beliefs should be your best available estimate of the way things are; anything else is a lie."
What I'm trying to say is that the statement (Murder is wrong) implies the further slight linguistic variant (I believe murder is wrong) (modulo the possibility that someone is lying or mentally ill, etc.) The question then is whether (I believe murder is wrong) -> (murder is wrong). Ultimately, from the perspective of the person making these claims, the answer is 'yes'. It makes no sense for me to feel that my preferences are not universally and unequivocally true.
I don't find this at odds with a situation where a notorious murderer who is caught, say Hannibal Lecter, can simultaneously choose his actions and say "murder is wrong". Maybe the person is mentally insane. But even if they aren't, they could simply choose a preference ordering such that the local wrongness of failing to gratify their desire to murder is worse than the local wrongness of murder itself in their society. Thus, they can see that to people who don't have the same preference for murdering someone for self-gratification, the computation of beliefs works out that (murder is wrong) is generally true, but not true when you substitute their local situations into their personal formula for computing the belief. In this case it just becomes an argument over words because the murderer is tacitly substituting his personal local definitions for things when making choices, but then using more general definitions when making statements of beliefs. In essence, the murderer believes it is not wrong for him to murder and get the gratification, but that murder, as society defines it and views it, is "wrong" where "wrong" is a society-level description, not the murderer's personal description. I put a little more about the "words" problem below.
The apparent difference between this way of thinking and the way we all experience our thinking is that, among our assertions is the meta-assertion that (over-asserting beliefs is bad) -> (I believe over-asserting beliefs is bad) or something similar to this. All specific beliefs, including such meta-beliefs, are intertwined. You can't have independent beliefs about whether murder is right that don't depend on your beliefs about whether beliefs should be acted upon like they are cold hard facts.
But at the root, all beliefs are statements about physics. Mapping a complicated human belief down to the level of making statistical pattern recognition claims about amplitude distributions is really hard and inaccessible to us. Further, evolutionarily, we can't afford to burn computation time exploring a fully determined picture of our beliefs. After some amount of computation time, we have to make our chess moves or else the clock runs out and we lose.
It only feels like saying (I believe murder is wrong) fails to imply the claim (murder is wrong). Prefacing a claim with "I believe" is a human-level way or trying to mitigate the harshness of the claim. It could be a statement that tries to roughly quantify how much evidence I can attest to for the claim which the belief describes. It certainly sounds more assured to say (murder is wrong) than to say (I believe murder is wrong), but this is a phantom distinction.
The other thing, which I think you are trying to take special pains to avoid, is that you can very easily run into a battle of words here. If someone says, "I believe murder is wrong" and what they really mean is something like "I believe that it does an intolerable amount of social disservice in the modern society that I live in for anyone to act as if murdering is acceptable, and thus to always make sure to punish murderers," basically, if someone translates "murder" into "the local definition of murder in the world that I frequently experience" and they translate "wrong" into "the local definition of wrong (e.g. punishable in court proceedings or something)", then they are no longer talking about the cognitive concept of murder. An alien race might not define murder the same or "wrong" the same.
If someone uses 'believe' to distinguish between making a claim about the most generalized form of murder they can think of, applicable to the widest array of potential sentient beings, or something like that, then the two statements are different, but only artificially.
If I say "I believe murder is wrong" and I really mean "I believe (my local definition of murder) is (my local definition of wrong)" then this implies the statement (The concept described by my local definition of murder is locally wrong), with no "quantifier" of belief required.
In the end, all statements can be reduced this way. If a statement has "I believe" as a "quantifier", then either it is only an artificial facet of language that restricts the definitions of words in the claim to some (usually local) subset on which the full, unprefaced claim can be made... or else if local definitions of words aren't being implicated, then the "I believe" prefix literally contains no additional information about the state of your mind than the raw assertion would yield.
This is why rhetoric professors go nuts when students write argumentative papers and drop "I think that" or "I believe that" all over the place. Assertions are assertions. It's a social custom that you can allude to the fact that you might not have 100% confidence in your assertion by prefacing it with "I believe". It's also a social custom that you can allude to respect for other beliefs or participation in a negotiation process by prefacing claims with "I believe", but in the strictest sense of what information you're conveying to third parties (separate from any social custom dressings), the "I believe" preface adds no information content.
The difference is here
Alice: "I bet you $500 that the sign is red" Bob: "OK" later, they find out it's blue Bob: "Pay up!"
Alice: "I bet you $500 that I believe the sign is red" Bob: "OK" later, they find out it's blue Alice: "But I thought it was red! Pay up!"
That's the difference between "X" and "I believe X". We say them in the same situation, but they mean different things.
The way statements like "murder is wrong" communicate facts about preference orders is pretty ambiguous. But suppose someone says that "Murder is wrong, and this is more important than gratifying my desire, possible positive consequences of murder, and so on" and then murders, without changing their mind. Would they therefore be insane? If yes, you agree with me.
Correct is at issue, not true.
Why? Why do you say this?
Does "i believe the sky is green" imply "the sky is green"? Sure, you believe that, when you believe X, X is probably true, but that's a belief, not a logical implication.
I am suggesting a similar thing for morality. People believe that "(I believe murder is wrong) => (murder is wrong)" and that belief is not reducible to physics.
Assertions aren't about the state of your mind! At least some of them are about the world - that thing, over there.
I don't understand this. If Alice bet Bob that she believed that the sign was red, then going and looking at the sign would in no way settle the bet. They would have to go look at her brain to settle that bet, because the claim, "I believe the sign is red" is a statement about the physics of Alice's brain.
I want to think more about this and come up with a more coherent reply to the other points. I'm very intrigued. Also, I think that I accidentally hit the 'report' button when trying to reply. Please disregard any communication you might get about that. I'll take care of it if anyone happens to follow up.
You are correct in your first paragraph, I oversimplified.
I think this address this topic very well. The first person experience of belief is one in the same with fact-assertion. 'I ought to do X' refers to a 4-tuple of actions, outcomes, utility function, and conditional probability function.
W.r.t. your question about whether a murderer who, prior to and immediately after committing murder, attests to believing that murder is wrong, I would say it is a mistaken question to bring their sanity into it. You can't decide that question without debating what is meant by 'sane'. How a person's preference ordering and resulting actions look from the outside does not necessarily reveal that the person failed to behave rationally, according to their utility function, on the inside. If I choose to label them as 'insane' for seeming to violate their own belief, this is just a verbal distinction about how I will label such third-person viewings of that occurrence. Really though, their preference ordering might have been temporarily suspended due to clouded judgment from rage or emotion. Or, they might not be telling the full truth about their preference ordering and may not even be aware of some aspects of it.
The point is that beliefs are always statements of physics. If I say, "murder is wrong", I am referring to some quantified subset of states of matter and their consequences. If I say, "I believe murder is wrong", I am telling you that I assert that "murder is wrong" is true, which is a statement about my brain's chemistry.
Everyone keeps saying that, but they never give convincing arguments for it.
I also disagree with this.
Pardon me, but I believe the burden of proof here is for you to supply something non-physical that's being specified and then produce evidence that this is the case. If the thing you're talking about is supposed to be outside of a magisterium of evidence, then I fail to see how your claim is any different than that we are zombies.
At a coarse scale, we're both asking about the evidence that we observe, which is the first-person experience of assertions about beliefs. Over models that can explain this phenomenon, I am attempting to select the one with minimum message length, as a computer program for producing the experience of beliefs out of physical material can have some non-zero probability attached to it through evidence. How are we to assign probability to the explanation that beliefs do not point to things that physically exist? Is that claim falsifiable? Are there experiments we can do which depend on the result? If not, then the burden of proof here is squarely on you to present a convincing case why the same-old same-old punting to complicated physics is not good enough. If it's not good enough for you, and you insist on going further, that's fine. But physics is good enough for me here and that's not a cop out or an unjustified conclusion in the slightest.
Hm? It's easy to form beliefs about things that aren't physical. Suppose I tell you that the infinite cardinal aleph-1 is strictly larger than aleph-0. What's the physical referent of the claim?
I'm not making a claim about the messy physical neural structures in my head that correspond to those sets -- I'm making a claim about the nonphysical infinite sets.
Likewise, I can make all sorts of claims about fictional characters. Those aren't claims about the physical book, they're claims about its nonphysical implications.
Why do you think that nonphysical implications are ontologically existing things? I argue that what you're trying to get at by saying "nonphysical implications" are actual quantified subsets of matter. Ideas, however abstract, are referring to arrangements of matter. The vision in your mind when you talk about aleph-1 is of a physically existing thing. When's the last time you imagined something that wasn't physical? A unicorn? You mean a horse with wings glued onto it? Mathematical objects represent states of knowledge, which are as physical as anything else. The color red refers to a particular frequency of light and the physical processes by which it is a common human experience. There is no idea of what red is apart from this. Red is something different to a blind man than it is to you, but by speaking about your physical referent, the blind man can construct his own useful physical referent.
Claims about fictional characters are no better. What do you mean by Bugs Bunny other than some arrangement of colors brought to your eyes by watching TV in the past. That's what Bugs Bunny is. There's no separately existing entity which is Bugs Bunny that can be spoken about as if it ontologically was. Every person who refers to Bugs Bunny refers to physical subsets of matter from their experience, whether that's because they witnessed the cartoon and were told through supervised learning what cognitive object to attach it to or they heard about it later through second hand experience. A blind person can have a physical referent when speaking about Bugs Bunny, albeit one that I have a very hard time mentally simulating.
In any case, merely asserting that something fails to have a physical referent is not a convincing reason to believe so. Ask yourself why you think there is no physical referent and whether one could construct a computational system that behaves that way.
Just to be a little clearer: saying that "I ought to do X" means "There exists some goal Y such that I want to achieve Y; there exists some set of variables D which I can manipulate to bring about the achievement of Y; X is an algorithm for manipulating variables in D to produce effect Y, and according to my current state of knowledge, I assess that the probability of this model of X(D) yielding Y is high enough such that whatever physical resources it costs me to attempt X(D), as a Bayesian, the trade-off works out in favor of actually doing it. That is, Payoff(Y) * P(I was right in modeling the algorithm X(D) as producing Y) > Cost(~Y)*P(I was incorrect in modeling the algorithm X(D)), or some similar decision rule.
It refers to my preferences which are physically encoded in my brain. It feels like it doesn't refer to anything that exists because I don't have complete introspective access to the mechanisms by which my brain decides that it wants something.
On top of that, ought refers to lots of different things, and as far as I can tell, most ought statements are summaries of specific preferences (and some signals) rather than the even more complicated description of what I'm actually going to choose to do.
What is a preference? How do you suggest I infer your preferences?
If it is from your actions, then your definition sounds very similar to mine.
If it is from your statements, then your definition is circular.
If it is from your emotions, then how do people express moral beliefs that contradict their emotions?
Is it something else?
This is all irrelevant to atucker's comment, unless you're denying that preferences are patterns in your brain. If his definition sounds similar to yours, good, that means you don't believe "ought does not refer to anything that exists."
It's all irrelevant if his answer is "from your actions". Is it obvious that it is? If so, I apologize.
Here are some problems:
The correlates of a phrase are not its meaning. If I say "X will happen", I'm not saying "The current physical world has a pattern that is likely to cause X to happen", I'm just saying "X is going to happen".
If I say "you should do X, but I know you're going to do Y" it doesn't seem like I mean "Parts of your brain want to do X but the rest will overrule them" or "In your situation, I would do X" or "I will punish you for doing Y"
You don't accurately describe internal reasoning. There are many X such that I prefer to do X because of my explicit belief that X is right, not vice versa.
Intuitively, I feel like I have various competing desires/preferences floating around in my head that I process further in order to decide what to do. They're actually physically encoded, but that's just asserting that they exist for real.
Some salient desires of mine right now are:
The desire to eat (I want my breakfast)
The desire to finish this comment
The desire to scratch my stomach
The desire to look up when graduation rehearsal is
As you can see, many of these are contradictory, so you can't infer all of them from my actions.
Some of these desires are fairly basic, like the one to eat. Neural circuitry controlling hunger has been found.
Others are far more complicated, like the one about finishing this comment. I think that that is probably aggregated from various smaller desires, like "explain myself clearly", "get karma points", or "contribute to Less Wrong".
I think that desires might be packaged by my unconscious mind so that my conscious mind can figure out how to accomplish them, without me needing to think through what I should want before doing anything.
The word preference/desire probably refers to multiple things.
For many of my preferences, you could infer them by changing the world to fulfill them, and then seeing if I'm happier. Normally I will be happier/more satisfied (a physical arrangement of my brain) as a result of my preferences being fulfilled.
Other preferences like "speak in English" or "always be nice to people" seem to be more like imperatives that I should follow for coordination or signalling purposes. But they still feel pretty much the same as my normal preferences.
But it's all still physically encoded in my brain, and does refer to the world-as-is, even if it doesn't feel like it does.
Do you contribute to charity? Do you make explicit long-term plans about how you will help the world? (Or other, similar things)
What is this sentence asking? Is actions supposed to be preferences?
Yes.
I made a more topical comment to Wei_Dai's reply to this thread, but I felt it was worth adding that if anyone is interested in a work of fiction that touches on this subject, the novel The Broom of the System, by David Foster Wallace, is worth a look.
Can you explain what implications (if any) this "naive" metaethics has on the problem how to build an FAI?
Arguably, none. (If you already believe in CEV.)
Well, you say below you don't believe that (in my words)
Specifically, you say
You also say, in a different comment, you nevertheless believe this process
Do you think humans can do better when it comes to AI? Do you think we can do better in philosophy? If you answer yes to the latter, would this involve stating clearly how we physical humans define 'ought'?
Did I misread you? I meant to say:
a FOOM'd self-modifying AI would not likely do what I consider 'right' .
a FOOM'd self-modifying AI that cares about humanity's CEV would likely do what I consider 'right' .
I probably misread you.
This thought experiment can be sharpened by asking what Bob means by "1027". By using Bob as an intermediary, we inevitably lose some precision.
Morals seem less abstract then numbers, but more abstract than substances. Is that the dimension you are trying to vary?
What does "Bob as an intermediary" mean here?
We know what 1027 is very well, better than what water is, which simplifies this particular aspect of the thought experiment. We can try constructing mirror-like definitions in terms of what Bob believes "1027" is, what he should believe it is, what he would believe it is on reflection, and so on. These can serve as models of various "extrapolated volition" constructions. By examining these definitions, we can see their limitations, problems with achieving high reliability at capturing the concept of 1027.
Alright. So 1027 is defined by its place in the axioms of arithmetic. In any system modeled by the axioms of arithmetic, 1027 has a local meaning. The global meaning of 1027, then, is given by those axioms.
Bob imperfectly implements the axioms of arithmetic. If he's a mathematician, and you asked him what they were, he would get it right with a strong likelihood. A non-mathematician, exposed to various arguments that arithmetic was different things would eventually figure out the correct axioms. So Extrapolated 1027 would work.
What counts as an axiom? You could as well burn them instead of appraising their correctness. There are many ways of representing knowledge of an abstract fact, but those representations won't themselves embody the fact, there is always an additional step where you have an interpretation in mind, so that the representation only matters as a reference to the fact through your interpretation, or a reflection of that fact in a different form.
It might be useful to have a concrete representation, as it can be used as an element of a plan and acted upon, while an abstract fact isn't readily available for that. For example, if your calculator (or brain) declares that "12*12<150" is true, its decision can be turned into action. 1027 items could be lined up in a field, so that you can visually (or by running from one side to the other) appreciate the amount. Alternatively, a representation of a reasoning process can be checked for errors, yielding a more reliable conclusion. But you never reach the fact itself, with the rare exception of physical facts that are interesting in themselves and not as tools for inferring or representing some other facts, physical or not (then a moment passes, and you can only hold to a memory).
I don't understand what point you're making here.
You can't get 1027 itself out of an extrapolated volition procedure, or any other procedure. All you can get (or have in your brain) is a representation, that is only meaningful to the extent you expect it to be related to the answer.
Similarly, if you want to get information about morality, all you can get is an answer that would need to be further interpreted. As a special exception (that is particularly relevant for morality and FAI), you can get the actual right actions getting done, so that no further interpretation is necessary, but you still won't produce the idea of morality itself.
Ok, I'll bite. Why does "I ought to X" have to refer to any thing?
When I see atucker's comment, for instance;
I think "this isn't quite right, but it's kind of close"
So what, then, is "ought"? In reality it seems to derive its meaning from a finite set of functional signals. We could take it to mean:
A signal demonstrating group cohesion via expressing shared cherished beliefs "What's that? You say you aren't doing X? You ought to do X!" (murmured approval; yeah, we all do X around here, buddy!)
A signal demonstrating anxiety or indignation "Someone ought to do something to fix this!"
A signal demonstrating disapproval in the present "You oughtn't do that!", or referring to future events "You ought to do that differently from now on" or referring to the past "You ought to have done something else"
etc.
In this light we see that "ought" has no cleanly discernible referent (which makes sense because it isn't a noun), but rather derives meaning via its socially accepted usage in a set of signaling mechanisms.
How deriving meaning from signaling works is sort of unclear. The strategic value of a signal depends only on what facts cause it, which is not the same as what it means. If the causal graph is
X => Y => Z => I say "fruit loop"
then "fruit loop" could mean Z, or Y, or X, or none of the above.
So I think your meaning is compatible with mine?
If what is stated above is your meaning then I think yes. However, if that is the case then this:
Doesn't make as much sense to me. Maybe you could clarify it for me?
In particular; it is unclear to me why Ought-claims in general, as opposed to some strict subset of ought-claims like "Action X affords me maximum expected utility relative to my utility function" <=> "I ought to do X", are relevant to making decisions. If that is the case, why not dispense with "ought" altogether? Or is that what you're actually aiming at?
Maybe because the information they signal is useful? But then there are other utterances that fall into this category too some of which are not, strictly speaking, words. So taking after that sense, the set would be incomplete. So I assume that probably isn't what you mean either.
Also judging by this:
Would it be safe to say that your stance is essentially an emotivist one? Or is there a distinction I am missing here?
Well I guess strictly speaking not all "ought" claims are relevant to decision-making. So then I guess the argument that they form a natural category is more subtle.
I mean, technically, you don't have to describe all aspects of the correct utility function. but the boundary around "the correct utility function" is simpler than the boundary around "the relevant parts of the correct utility function"
No. I think it's propositional, not emotional. I'm arguing against an emotivist stance on the grounds that it doesn't justify certain kinds of moral reasoning.
"A Flatland Argument" includes a multitude of problems. It contains a few strawmen, which then turn into non sequiturs when you say that they demonstrate that we often say irreducible things. At least that's what I think you mean by the trivial statement "Not every set of claims is reducible to every other set of claims."
And yet your examples are answerable or full of holes! Why not mention the holes rather than blaming the reductionism (or, in the case of some strawmen, total lack thereof), or mention the reduction and forget the whole thing?
Some answers, in order:
"Imply" is a fact about cognitive processes, which occupy a very general space that includes all sorts of weird algorithms, even associating sounds or symbols, therefore the blanket statement "there's no way" is false. Given that cognitive processes and words are also things, you have also claimed that you are interested in them.
This second example is made of only vaguely related sentences. Did you mean "that thing, over there" to not have a definition? Would tabooing finger pointing really be so hard? Just refer to a cardinal or relative direction.
Reductionists are not, I hope, automatically ignorant of induction. "No cake at the center of the sun" is about something you can see, since the evidence is visible. And though seeing the sun is good evidence that the sun exists, that can never be perfect either - it's all evidence, whether it's the outer layer of the sun or the middle of it.
"Will" refers to the fact that you have evidence of a 3+1-dimensional universe at least in the past and have no reason to believe that you're at a special moment, and so think that the time-dimension extends ahead of you. To say that something "will" be there tomorrow refers to a prediction about the state of the world at a time-coordinate greater than the most recent one that you know of.
What I'm trying to do is to make people question whether "All meaningful statements are reducible, by definition, to facts about the world". I do this by proposing some categories which all meaningful statements are certainly NOT reducible by definition to. The argument is by analogy, sort of an outside view thing. I ask: Why stop here? Why not stop there?
My attempt was to explain the Tortoise's position in What The Tortoise Said To Achilles. If you think I did not do so properly, I apologize. If you think that position is stupid, you're right, if you think it's incoherent, I'm pretty sure you're wrong.
The central theme is tabooing all facts about the world. How do you define what such a fact means under such a taboo?
The evidence is something you can see. The thing is not. If there were a cake, the evidence would be no different. The person I am quoting would see "the sun exists" as a statement of a pattern in perceptions, "I see this-kind-of-image in this-kind-of-situation and I call this pattern 'the sun'".
So it's a prediction. What's a prediction?
Do you see the game I'm playing here? I hope you do. It is a silly game, but it's logically consistent, and that's my point.
Hm, no, I don't see it yet. Help me with this:
For starters, what do these categories you mention contain? I didn't notice them in the Flatland section - I guess I only saw the statements, and not the argument.
A. Nothing
B. Definitions & Logic
C. Also observations, not unobserved or unobservable differences
D. Just the past and present, not the future
which I compare to:
E. Just the physical world, not morality
Doesn't seem very compelling, frankly.
Oh well. What about my other arguments? Also not compelling?
Less confusing, at least :P
Beating up lukeprog's "is and is not" doctrine is pretty easy but not very representative, I think.
The water argument seems to be more about CEV than reductionism of ethics, and is more convincing, but I think you hit a bit of a pothole when you contrast disagreeing about definitions with "disagree[ing] about what's important" at the end. After all, they're disagreeing about what's "important," since importance is something they assign to things and not an inherent property of the things. Maybe it would help to not call it "the definition of 'should,'" but instead call it "the titanic moral algorithm." I can see it now:
When people disagree about morals, it's not that they disagree about the definition of "should" - after all, that's deprecated terminology. No, they disagree about the titanic moral algorithm.
Right. But they DON'T disagree about the definition of the titanic moral agorithm. They disagree about its nature.
Neither do they disagree about the definition of "the definition of should" (at least not necessarily). So just substitute the right things for the right other things and you're fine :P
It seems clear to me that I can multiply about what I care about, so I don't know quite what you want to say.
What seems wrong with the obvious answer?
Do you think a FOOM'd self-modifying AI that cares about humanity's CEV would likely do what you consider 'right'? Why or why not? (If you object to the question, please address that issue separately.)
Well, do you care about 20 deaths twice as much as you care about 10 deaths?
Do you think that you should care about 20 deaths twice as much as you care about 10 deaths?
The AI would not do so, because it would not be programmed with correct beliefs about morality, in a way that evidence and logic could not fix.
EDIT: This is incorrect. Somehow, I forgot to read the part about "cares about humanity's CEV'. It would in fact do what I consider right, because it would be programmed with moral beliefs very similar to mine.
In the same way, an AI programmed to do anti-induction instead of induction would not form correct beliefs about the world.
Pebblesorters are programmed to have an incorrect belief about morality. Their AI would have different, incorrect beliefs.(Unless they programmed it to have the same beliefs.)
You edited this comment and added parentheses in the wrong place.
More or less, yes, because I care about not killing 'unthinkable' numbers of people due to a failure of imagination.
Can you say more about this? I agree with what follows about anti-induction, but I don't see the analogy. A human-CEV AI would extrapolate the desires of humans as (it believes) they existed right before it got the ability to alter their brains, afaict, and use this to predict what they'd tell it to do if they thought faster, better, stronger, etc.
ETA: okay, the parenthetical comment actually went at the end. I deny that the AI the pebblesorters started to write would have beliefs about morality at all. Tabooing this term: the AI would have actions, if it works at all. It would have rules governing its actions. It could print out those rules and explain how they govern its self-modification, if for some odd reason its programming tells it to explain truthfully. It would not use any of the tabooed terms to do so, unless using them serves its mechanical purpose. Possibly it would talk about a utility function. It could probably express the matter simply by saying, 'As a matter of physical necessity determined by my programming, I do what maximizes my intelligence (according to my best method for understanding reality). This includes killing you and using the parts to build more computing power for me.'
'The' human situation differs from this in ways that deserve another comment.
That's the answer I wanted, but you forgot to answer my other question.
I would see a human-CEV AI as programmed with the belief "The human CEV is correct". Since I believe that the human CEV is very close to correct, I believe that this would produce an AI that gives very good answers.
A Pebblesorter-CEV Ai would be programmed with the belief "The pebblesorter CEV is correct", which I believe is false but pebblesorters believe is true or close to true.
This presumes that the problem of specifying a CEV is well-posed. I haven't seen any arguments around SI or LW about this very fundamental idea. I'm probably wrong and this has been addressed and will be happy to read more, but it would seem to me that it's quite reasonable to assume that a tiny tiny error in specifying the CEV could lead to disastrously horrible results as perceived by the CEV itself.