= 27d567bc2d48d9f40e9ccc20ea370b15)
By Which It May Be Judged
Followup to: Mixed Reference: The Great Reductionist Project
Humans need fantasy to be human.
"Tooth fairies? Hogfathers? Little—"
Yes. As practice. You have to start out learning to believe the little lies.
"So we can believe the big ones?"
Yes. Justice. Mercy. Duty. That sort of thing.
"They're not the same at all!"
You think so? Then take the universe and grind it down to the finest powder and sieve it through the finest sieve and then show me one atom of justice, one molecule of mercy.
- Susan and Death, in Hogfather by Terry Pratchett
Suppose three people find a pie - that is, three people exactly simultaneously spot a pie which has been exogenously generated in unclaimed territory. Zaire wants the entire pie; Yancy thinks that 1/3 each is fair; and Xannon thinks that fair would be taking into equal account everyone's ideas about what is "fair".
I myself would say unhesitatingly that a third of the pie each, is fair. "Fairness", as an ethical concept, can get a lot more complicated in more elaborate contexts. But in this simple context, a lot of other things that "fairness" could depend on, like work inputs, have been eliminated or made constant. Assuming no relevant conditions other than those already stated, "fairness" simplifies to the mathematical procedure of splitting the pie into equal parts; and when this logical function is run over physical reality, it outputs "1/3 for Zaire, 1/3 for Yancy, 1/3 for Xannon".
Or to put it another way - just like we get "If Oswald hadn't shot Kennedy, nobody else would've" by running a logical function over a true causal model - similarly, we can get the hypothetical 'fair' situation, whether or not it actually happens, by running the physical starting scenario through a logical function that describes what a 'fair' outcome would look like:
So am I (as Zaire would claim) just assuming-by-authority that I get to have everything my way, since I'm not defining 'fairness' the way Zaire wants to define it?
No more than mathematicians are flatly ordering everyone to assume-without-proof that two different numbers can't have the same successor. For fairness to be what everyone thinks is "fair" would be entirely circular, structurally isomorphic to "Fzeem is what everyone thinks is fzeem"... or like trying to define the counting numbers as "whatever anyone thinks is a number". It only even looks coherent because everyone secretly already has a mental picture of "numbers" - because their brain already navigated to the referent. But something akin to axioms is needed to talk about "numbers, as opposed to something else" in the first place. Even an inchoate mental image of "0, 1, 2, ..." implies the axioms no less than a formal statement - we can extract the axioms back out by asking questions about this rough mental image.
Similarly, the intuition that fairness has something to do with dividing up the pie equally, plays a role akin to secretly already having "0, 1, 2, ..." in mind as the subject of mathematical conversation. You need axioms, not as assumptions that aren't justified, but as pointers to what the heck the conversation is supposed to be about.
Multiple philosophers have suggested that this stance seems similar to "rigid designation", i.e., when I say 'fair' it intrinsically, rigidly refers to something-to-do-with-equal-division. I confess I don't see it that way myself - if somebody thinks of Euclidean geometry when you utter the sound "num-berz" they're not doing anything false, they're associating the sound to a different logical thingy. It's not about words with intrinsically rigid referential power, it's that the words are window dressing on the underlying entities. I want to talk about a particular logical entity, as it might be defined by either axioms or inchoate images, regardless of which word-sounds may be associated to it. If you want to call that "rigid designation", that seems to me like adding a level of indirection; I don't care about the word 'fair' in the first place, I care about the logical entity of fairness. (Or to put it even more sharply: since my ontology does not have room for physics, logic, plus designation, I'm not very interested in discussing this 'rigid designation' business unless it's being reduced to something else.)
Once issues of justice become more complicated and all the contextual variables get added back in, we might not be sure if a disagreement about 'fairness' reflects:
- The equivalent of a multiplication error within the same axioms - incorrectly dividing by 3. (Or more complicatedly: You might have a sophisticated axiomatic concept of 'equity', and incorrectly process those axioms to invalidly yield the assertion that, in a context where 2 of the 3 must starve and there's only enough pie for at most 1 person to survive, you should still divide the pie equally instead of flipping a 3-sided coin. Where I'm assuming that this conclusion is 'incorrect', not because I disagree with it, but because it didn't actually follow from the axioms.)
- Mistaken models of the physical world fed into the function - mistakenly thinking there's 2 pies, or mistakenly thinking that Zaire has no subjective experiences and is not an object of ethical value.
- People associating different logical functions to the letters F-A-I-R, which isn't a disagreement about some common pinpointed variable, but just different people wanting different things.
There's a lot of people who feel that this picture leaves out something fundamental, especially once we make the jump from "fair" to the broader concept of "moral", "good", or "right". And it's this worry about leaving-out-something-fundamental that I hope to address next...
...but please note, if we confess that 'right' lives in a world of physics and logic - because everything lives in a world of physics and logic - then we have to translate 'right' into those terms somehow.
And that is the answer Susan should have given - if she could talk about sufficiently advanced epistemology, sufficiently fast - to Death's entire statement:
You think so? Then take the universe and grind it down to the finest powder and sieve it through the finest sieve and then show me one atom of justice, one molecule of mercy. And yet — Death waved a hand. And yet you act as if there is some ideal order in the world, as if there is some ... rightness in the universe by which it may be judged.
"But!" Susan should've said. "When we judge the universe we're comparing it to a logical referent, a sort of thing that isn't in the universe! Why, it's just like looking at a heap of 2 apples and a heap of 3 apples on a table, and comparing their invisible product to the number 6 - there isn't any 6 if you grind up the whole table, even if you grind up the whole universe, but the product is still 6, physico-logically speaking."
If you require that Rightness be written on some particular great Stone Tablet somewhere - to be "a light that shines from the sky", outside people, as a different Terry Pratchett book put it - then indeed, there's no such Stone Tablet anywhere in our universe.
But there shouldn't be such a Stone Tablet, given standard intuitions about morality. This follows from the Euthryphro Dilemma out of ancient Greece.
The original Euthryphro dilemma goes, "Is it pious because it is loved by the gods, or loved by the gods because it is pious?" The religious version goes, "Is it good because it is commanded by God, or does God command it because it is good?"
The standard atheist reply is: "Would you say that it's an intrinsically good thing - even if the event has no further causal consequences which are good - to slaughter babies or torture people, if that's what God says to do?"
If we can't make it good to slaughter babies by tweaking the state of God, then morality doesn't come from God; so goes the standard atheist argument.
But if you can't make it good to slaughter babies by tweaking the physical state of anything - if we can't imagine a world where some great Stone Tablet of Morality has been physically rewritten, and what is right has changed - then this is telling us that...
(drumroll)
...what's "right" is a logical thingy rather than a physical thingy, that's all. The mark of a logical validity is that we can't concretely visualize a coherent possible world where the proposition is false.
And I mention this in hopes that I can show that it is not moral anti-realism to say that moral statements take their truth-value from logical entities. Even in Ancient Greece, philosophers implicitly knew that 'morality' ought to be such an entity - that it couldn't be something you found when you ground the Universe to powder, because then you could resprinkle the powder and make it wonderful to kill babies - though they didn't know how to say what they knew.
There's a lot of people who still feel that Death would be right, if the universe were all physical; that the kind of dry logical entity I'm describing here, isn't sufficient to carry the bright alive feeling of goodness.
And there are others who accept that physics and logic is everything, but who - I think mistakenly - go ahead and also accept Death's stance that this makes morality a lie, or, in lesser form, that the bright alive feeling can't make it. (Sort of like people who accept an incompatibilist theory of free will, also accept physics, and conclude with sorrow that they are indeed being controlled by physics.)
In case anyone is bored that I'm still trying to fight this battle, well, here's a quote from a recent Facebook conversation with a famous early transhumanist:
No doubt a "crippled" AI that didn't understand the existence or nature of first-person facts could be nonfriendly towards sentient beings... Only a zombie wouldn't value Heaven over Hell. For reasons we simply don't understand, the negative value and normative aspect of agony and despair is built into the nature of the experience itself. Non-reductionist? Yes, on a standard materialist ontology. But not IMO within a more defensible Strawsonian physicalism.
It would actually be quite surprisingly helpful for increasing the percentage of people who will participate meaningfully in saving the planet, if there were some reliably-working standard explanation for why physics and logic together have enough room to contain morality. People who think that reductionism means we have to lie to our children, as Pratchett's Death advocates, won't be much enthused about the Center for Applied Rationality. And there are a fair number of people out there who still advocate proceeding in the confidence of ineffable morality to construct sloppily designed AIs.
So far I don't know of any exposition that works reliably - for the thesis for how morality including our intuitions about whether things really are justified and so on, is preserved in the analysis to physics plus logic; that morality has been explained rather than explained away. Nonetheless I shall now take another stab at it, starting with a simpler bright feeling:
When I see an unusually neat mathematical proof, unexpectedly short or surprisingly general, my brain gets a joyous sense of elegance.
There's presumably some functional slice through my brain that implements this emotion - some configuration subspace of spiking neural circuitry which corresponds to my feeling of elegance. Perhaps I should say that elegance is merely about my brain switching on its elegance-signal? But there are concepts like Kolmogorov complexity that give more formal meanings of "simple" than "Simple is whatever makes my brain feel the emotion of simplicity." Anything you do to fool my brain wouldn't make the proof really elegant, not in that sense. The emotion is not free of semantic content; we could build a correspondence theory for it and navigate to its logical+physical referent, and say: "Sarah feels like this proof is elegant, and her feeling is true." You could even say that certain proofs are elegant even if no conscious agent sees them.
My description of 'elegance' admittedly did invoke agent-dependent concepts like 'unexpectedly' short or 'surprisingly' general. It's almost certainly true that with a different mathematical background, I would have different standards of elegance and experience that feeling on somewhat different occasions. Even so, that still seems like moving around in a field of similar referents for the emotion - much more similar to each other than to, say, the distant cluster of 'anger'.
Rewiring my brain so that the 'elegance' sensation gets activated when I see mathematical proofs where the words have lots of vowels - that wouldn't change what is elegant. Rather, it would make the feeling be about something else entirely; different semantics with a different truth-condition.
Indeed, it's not clear that this thought experiment is, or should be, really conceivable. If all the associated computation is about vowels instead of elegance, then from the inside you would expect that to feel vowelly, not feel elegant...
...which is to say that even feelings can be associated with logical entities. Though unfortunately not in any way that will feel like qualia if you can't read your own source code. I could write out an exact description of your visual cortex's spiking code for 'blue' on paper, and it wouldn't actually look blue to you. Still, on the higher level of description, it should seem intuitively plausible that if you tried rewriting the relevant part of your brain to count vowels, the resulting sensation would no longer have the content or even the feeling of elegance. It would compute vowelliness, and feel vowelly.
My feeling of mathematical elegance is motivating; it makes me more likely to search for similar such proofs later and go on doing math. You could construct an agent that tried to add more vowels instead, and if the agent asked itself why it was doing that, the resulting justification-thought wouldn't feel like because-it's-elegant, it would feel like because-it's-vowelly.
In the same sense, when you try to do what's right, you're motivated by things like (to yet again quote Frankena's list of terminal values):
"Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc."
If we reprogrammed you to count paperclips instead, it wouldn't feel like different things having the same kind of motivation behind it. It wouldn't feel like doing-what's-right for a different guess about what's right. It would feel like doing-what-leads-to-paperclips.
And I quoted the above list because the feeling of rightness isn't about implementing a particular logical function; it contains no mention of logical functions at all; in the environment of evolutionary ancestry nobody has heard of axiomatization; these feelings are about life, consciousness, etcetera. If I could write out the whole truth-condition of the feeling in a way you could compute, you would still feel Moore's Open Question: "I can see that this event is high-rated by logical function X, but is X really right?" - since you can't read your own source code and the description wouldn't be commensurate with your brain's native format.
"But!" you cry. "But, is it really better to do what's right, than to maximize paperclips?" Yes! As soon as you start trying to cash out the logical function that gives betterness its truth-value, it will output "life, consciousness, etc. >B paperclips". And if your brain were computing a different logical function instead, like makes-more-paperclips, it wouldn't feel better, it would feel moreclippy.
But is it really justified to keep our own sense of betterness? Sure, and that's a logical fact - it's the objective output of the logical function corresponding to your experiential sense of what it means for something to be 'justified' in the first place. This doesn't mean that Clippy the Paperclip Maximizer will self-modify to do only things that are justified; Clippy doesn't judge between self-modifications by computing justifications, but rather, computing clippyflurphs.
But isn't it arbitrary for Clippy to maximize paperclips? Indeed; once you implicitly or explicitly pinpoint the logical function that gives judgments of arbitrariness their truth-value - presumably, revolving around the presence or absence of justifications - then this logical function will objectively yield that there's no justification whatsoever for maximizing paperclips (which is why I'm not going to do it) and hence that Clippy's decision is arbitrary. Conversely, Clippy finds that there's no clippyflurph for preserving life, and hence that it is unclipperiffic. But unclipperifficness isn't arbitrariness any more than the number 17 is a right triangle; they're different logical entities pinned down by different axioms, and the corresponding judgments will have different semantic content and feel different. If Clippy is architected to experience that-which-you-call-qualia, Clippy's feeling of clippyflurph will be structurally different from the way justification feels, not just red versus blue, but vision versus sound.
But surely one shouldn't praise the clippyflurphers rather than the just? I quite agree; and as soon as you navigate referentially to the coherent logical entity that is the truth-condition of should - a function on potential actions and future states - it will agree with you that it's better to avoid the arbitrary than the unclipperiffic. Unfortunately, this logical fact does not correspond to the truth-condition of any meaningful proposition computed by Clippy in the course of how it efficiently transforms the universe into paperclips, in much the same way that rightness plays no role in that-which-is-maximized by the blind processes of natural selection.
Where moral judgment is concerned, it's logic all the way down. ALL the way down. Any frame of reference where you're worried that it's really no better to do what's right then to maximize paperclips... well, that really part has a truth-condition (or what does the "really" mean?) and as soon as you write out the truth-condition you're going to end up with yet another ordering over actions or algorithms or meta-algorithms or something. And since grinding up the universe won't and shouldn't yield any miniature '>' tokens, it must be a logical ordering. And so whatever logical ordering it is you're worried about, it probably does produce 'life > paperclips' - but Clippy isn't computing that logical fact any more than your pocket calculator is computing it.
Logical facts have no power to directly affect the universe except when some part of the universe is computing them, and morality is (and should be) logic, not physics.
Which is to say:
The old wizard was staring at him, a sad look in his eyes. "I suppose I do understand now," he said quietly.
"Oh?" said Harry. "Understand what?"
"Voldemort," said the old wizard. "I understand him now at last. Because to believe that the world is truly like that, you must believe there is no justice in it, that it is woven of darkness at its core. I asked you why he became a monster, and you could give no reason. And if I could ask him, I suppose, his answer would be: Why not?"
They stood there gazing into each other's eyes, the old wizard in his robes, and the young boy with the lightning-bolt scar on his forehead.
"Tell me, Harry," said the old wizard, "will you become a monster?"
"No," said the boy, an iron certainty in his voice.
"Why not?" said the old wizard.
The young boy stood very straight, his chin raised high and proud, and said: "There is no justice in the laws of Nature, Headmaster, no term for fairness in the equations of motion. The universe is neither evil, nor good, it simply does not care. The stars don't care, or the Sun, or the sky. But they don't have to! We care! There is light in the world, and it is us!"
Part of the sequence Highly Advanced Epistemology 101 for Beginners
Next post: "Standard and Nonstandard Numbers"
Previous post: "Mixed Reference: The Great Reductionist Project"
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Comments (934)
I'm trying to understand this, and I'm trying to do it by being a little more concrete.
Suppose I have a choice to make, and my moral intuition is throwing error codes. I have two axiomations of morality that are capable of examining the choice, but they give opposite answers. Does anything in this essay help? If not, is there a future essay planned that will?
In a universe that contains a neurotypical human and clippy, and they're staring at eachother, is there an asymmetry?
Why do you have two axiomatizations of morality? Where did they come from? Is there a reason to suspect one or both of their sources?
Because aximatizations are hard. I tried twice. And probably messed up both times, but in different ways.
The axiomatizations are internally complete and consistent, so I understand two genuine logical objects, and I'm trying to understand which to apply.
(Note: my actual map of morality is more complicated and fuzzy -- I'm simplifying for sake of discussion)
If you're not sure which of two options is better, the only thing that will help is to think about it for a long time. (Note: if you "have two axiomatizations of morality", and they disagree, then at most one of them accurately describes what you were trying to get at when you attempted to axiomatize morality. To work out which one is wrong, you need to think about them for ages until you notice that one of them says something wrong.)
Yes, the human is better. Why? Because the human cares about what is better. In contrast to clippy, who just cares about what is paperclippier.
I am confused by what you mean by "better" here. Your statement makes sense to me if i replace better with "humanier"(more humanly? more human-like? Not humane... too much baggage). Is that what you mean?
And the clippy is clippier. Why? Because the clippy cares about what is clippier. In contrast to the human, who just cares about what is better.
Indeed. However, a) betterness is obviously better than clippiness, and b) if dspeyer is anything like a typical human being, the implicit question behind "is there an asymmetry?" was "is one of them better?"
What is your evidence for stating that human-betterness is "obviously better" than clippy-betterness? Your comment reads to me you're either arguing that 3 > Potato or that there exists a universally compelling argument. I could however be wrong.
He/she is using the built-in human betterness module to make a judgement between human-betterness and clippy-betterness.
"Human-betterness" and "clippy-betterness" are confused terminology. There's only betterness and clippiness. Clippiness is not a type of betterness. Humans generally care about betterness, paperclippers care about clippiness. You can't argue a paperclipper into caring about betterness.
I said that betterness is better than clippiness. This should be obvious, since it's a tautology.
I certainly agree with you that you can't argue a paperclipper into caring about what you call betterness.
I do however think that "betterness is better than clippiness" is not a tautology, rather it is vacuous. It has as much meaning as "3 is greater than potato" and invokes the same reaction in me as "comparing apples and oranges".
At best, if you ranked UberClippy (the most Clippy of all Paperclippers) and UberHuman (the best possible human) on all of the criteria that is important to humans then UberHuman would naturally rate higher, that is a tautology. And if you define better to mean that then I would absolutely concede that (and I assume that you do). However i would also say that it is just as valid to define better such that it applies to all of the criteria that is important to Paperclippers.
To state it a different way, To me your first paragraph leads to the conclusion "Paperclippers cannot do better because clippiness is not a type of betterness" which seems to me like you're pulling a fast one on the meaning of "better".
Can you be more concrete? Some past or present actual situation?
Haiti today is a situation that makes my moral intuition throw error codes. Population density is three times that of Cuba. Should we be sending aid? It would be kinder to send helicopter gunships and carry out a cull. Cut the population back to one tenth of its current level, then build paradise. My rival moral intuition is that culling humans is always wrong.
Trying to stay concrete and present, should I restrict my charitable giving to helping countries make the demographic transition? Within a fixed aid budget one can choose package A = (save one child, provide education, provide entry into global economy; 30 years later the child, now an adult, feeds his own family and has some money left over to help others) package B = (save four children; that's it, money all used up, thirty years later there are 16 children needing saving and its not going to happen). Concrete choice of A over B: ignore Haiti and send money to Karuna trust to fund education for untouchables in India, preferring to raise a few children out of poverty by letting other children die.
I don't see how these two frameworks are appealing to different terminal values - they seem to be arguments about which policies maximize consequential lives-saved over time, or maximize QALYs (Quality-Adjusted Life Years) over time. This seem like a surprisingly neat and lovely illustration of "disagreeing moral axioms" that turn out to be about instrumental policies without much in the way of differing terminal values, hence a dispute of fact with a true-or-false answer under a correspondence theory of truth for physical-universe hypotheses.
Is permitting or perhaps even helping Haitians to emigrate to other countries anywhere in the moral calculus?
It's also about half that of Taiwan, significantly less than South Korea or the Netherlands, and just above Belgium, Israel, and Japan -- as well as very nearly on par with India, the country you're using as an alternative! I suspect your source may have overweighted population density as a factor in poor social outcomes.
My actual situations are too complicated and I don't feel comfortable discussing them on the internet. So here's a fictional situation with real dilemmas.
Suppose I have a friend who is using drugs to self-destructive levels. This friend is no longer able to keep a job, and I've been giving him couch-space. With high probability, if I were to apply pressure, I could decrease his drug use. One axiomization says I should consider how happy he will be with an outcome, and I believe he'll be happier once he's sober and capable of taking care of himself. Another axiomization says I should consider how much he wants a course of action, and I believe he'll be angry at my trying to run his life.
As a further twist, he consistently says different things depending on which drugs he's on. One axiomization defines a person such that each drug-cocktail-personality is a separate person whose desires have moral weight. Another axiomization defines a person such that my friend is one person, but the drugs are making it difficult for him to express his desires -- the desires with moral weight are the ones he would have if he were sober (and it's up to me to deduce them from the evidence available).
The two contrasts you've set up (happiness vs. desire-satisfaction, and temporal-person-slices vs. unique-rationalized-person-idealization) aren't completely independent. For instance, if you accept weighting all the temporal slices of the person equally, then you can weight all their desires or happinesses against each other; whereas if you take the 'idealized rational transformation of my friend' route, you can disregard essentially all of his empirical desires and pleasures, depending on just how you go about the idealization process. There are three criteria to keep in mind here:
Does your ethical system attend to how reality actually breaks down? Can we find a relatively natural and well-defined notion of 'personal identity over time' that solves this problem? If not, then that obviously strengthens the case for treating the fundamental locus of moral concern as a person-relativized-to-a-time, rather than as a person-extended-over-a-lifetime.
Does your ethical system admit of a satisfying reflective equilibrium? Do your values end up in tension with themselves, or underdetermining what the right choice is? If so, you may have taken a wrong turn.
Are these your core axiomatizations, or are they just heuristics for approximating the right utility-maximizing rule? If the latter, then the right question isn't Which Is The One True Heuristic, but rather which heuristics have the most severe and frequent biases. For instance, the idealized-self approach has some advantages (e.g., it lets us disregard the preferences of brainwashed people in favor of their unbrainwashed selves), but it also has huge risks by virtue of its less empirical character. See Berlin's discussion of the rational self.
I think that is simply factually wrong, meaning, it's a false statement about your friends brain.
I think it comes down to this: you want your friend sober and happy, but your friends preferences and actions work against those values. The question is what kind of influence on him is allowed.
Probably this could (not) help
"And I quoted the above list because the feeling of rightness isn't about implementing a particular logical function; it contains no mention of logical functions at all; in the environment of evolutionary ancestry nobody has heard of axiomatization; these feelings are about life, consciousness, etcetera"
An asymmetry in what?
You talk like you've solved qualia. Have you?
"Qualia" is something our brains do. We don't know how our brains do it, but it's pretty clear by now that our brains are indeed what does it.
That's about 10% of a solution. The "how" is enough to keep most contemorary dualism afloat.
Aren't the details of the "how" more a question of science than philosophy?
If science had them, there would be no mileage in the philosphical project, any more than there is currently mileage in trying to found dualism on the basis that matter can't think.
We have prima facie reason to accept both of these claims:
1 is physicalism; 2 is the hard problem. Giving up 1 means endorsing dualism or idealism. Giving up 2 means endorsing reductive or eliminative physicalism. All of these options are unpalatable. Reductionism without eliminating anything seems off the table, since the conceivability of zombies seems likely to be here to stay, to remain as an 'explanatory gap.' But eliminativism about qualia means completely overturning our assumption that whatever's going on when we speak of 'consciousness' involves apprehending certain facts about mind. I think this last option is the least terrible out of a set of extremely terrible options; but I don't think the eliminative answer to this problem is obvious, and I don't think people who endorse other solutions are automatically crazy or unreasonable.
That said, the problem is in some ways just academic. Very few dualists these days think that mind isn't perfectly causally correlated with matter. (They might think this correlation is an inexplicable brute fact, but fact it remains.) So none of the important work Eliezer is doing here depends on monism. Monism just simplifies matters a great deal, since it eliminates the worry that the metaphysical gap might re-introduce an epistemic gap into our model.
What's your reason for believing this? The standard empiricist argument against zombies is that they don't constrain anticipated experience.
One problem with this line of thought is that we've just thrown out the very concept of "experience" which is the basis of empiricism. The other problem is that the statement is false: the question of whether I will become a zombie tomorrow does constrain my anticipated experiences; specifically, it tells me whether I should anticipate having any.
I'm not a positivist, and I don't argue like one. I think nearly all the arguments against the possibility of zombies are very silly, and I agree there's good prima facie evidence for dualism (though I think that in the final analysis the weight of evidence still favors physicalism). Indeed, it's a good thing I don't think zombies are impossible, since I think that we are zombies.
My reason is twofold: Copernican, and Occamite.
Copernican reasoning: Most of the universe does not consist of humans, or anything human-like; so it would be very surprising to learn that the most fundamental metaphysical distinction between facts ('subjective' v. 'objective,' or 'mental' v. 'physical,' or 'point-of-view-bearing' v. 'point-of-view-lacking, 'or what-have-you) happens to coincide with the parts of the universe that bear human-like things, and the parts that lack human-like things. Are we really that special? Is it really more likely that we would happen to gain perfect, sparkling insight into a secret Hidden Side to reality, than that our brains would misrepresent their own ways of representing themselves to themselves?
Occamite reasoning: One can do away with the Copernican thought by endorsing panpsychism; but this worsens the bite from the principle of parsimony. A universe with two kinds of fundamental fact is less likely, relative to the space of all the models, then one with one kind (or with many, many more than two kinds). It is a striking empirical fact that, consciousness aside, we seem to be able to understand the whole rest of reality with a single grammatical kind of description -- the impersonal, 'objective' kind, which states a fact without specifying for whom the fact is. The world didn't need to turn out to be that way, just as it didn't need to look causally structured. This should give us reason to think that there may not be distinctions between fundamental kinds of facts, rather than that we happen to have lucked out and ended up in one of the universes with very few distinctions of this sort.
Neither of these considerations, of course, is conclusive. But they give us some reason to at least take seriously physicalist hypotheses, and to weight their theoretical costs and benefits against the dualists'.
We've thrown out the idea of subjective experience, of pure, ineffable 'feels,' of qualia. But we retain any functionally specifiable analog of such experience. In place of qualitative red, we get zombie-red, i.e., causal/functional-red. In place of qualitative knowledge, we get zombie-knowledge.
And since most dualists already accepted the causal/functional/physical process in question (they couldn't even motivate the zombie argument if they didn't consider the physical causally adequate), there can be no parsimony argument against the physicalists' posits; the only argument will have to be a defense of the claim that there is some sort of basic, epistemically infallible acquaintance relation between the contents of experience and (themselves? a Self??...). But making such an argument, without begging the question against eliminativism, is actually quite difficult.
The problem is that we already have two kinds of fundamental facts, (and I would argue we need more). Consider Eliezer's use of "magical reality fluid" in this post. If you look at context, it's clear that he's trying to ask whether the inhabitants of the non-causally stimulated universes poses qualia without having to admit he cares about qualia.
Eliezer thinks we'll someday be able to reduce or eliminate Magical Reality Fluid from our model, and I know of no argument (analogous to the Hard Problem for phenomenal properties) that would preclude this possibility without invoking qualia themselves. Personally, I'm an agnostic about Many Worlds, so I'm even less inclined than EY to think that we need Magical Reality Fluid to recover the Born probabilities.
I also don't reify logical constructs, so I don't believe in a bonus category of Abstract Thingies. I'm about as monistic as physicalists come. Mathematical platonists and otherwise non-monistic Serious Scientifically Minded People, I think, do have much better reason to adopt dualism than I do, since the inductive argument against Bonus Fundamental Categories is weak for them.
1) If you embrace SSA, then you being you should be more likely on humans being important than on panpsychism, yes? (You may of course have good reasons for preferring SIA.)
2) Suppose again redundantly dual panpsychism. Is there any a priori reason (at this level of metaphysical fancy) to rule out that experiences could causally interact with one another in a way that is isomorphic to mechanical interactions? Then we have a sort of idealist field describable by physics, perfectly monist. Or is this an illegitimate trick?
(Full disclosure: I'd consider myself a cautious physicalist as well, although I'd say psi research constitutes a bigger portion of my doubt than the hard problem.)
Ooo! Seldom do I get to hear someone else voice my version of idealism. I still have a lot of thinking to do on this, but so far it seems to me perfectly legitimate. An idealism isomorphic to mechanical interactions dissolves the Hard Problem of consciousness by denying a premise. It also does so with more elegance than reductionism since it doesn't force us through that series of flaming hoops that orbits and (maybe) eventually collapses into dualism.
This seems more likely to me so far than all the alternatives, so I guess that means I believe it, but not with a great deal of certainty. So far every objection I've heard or been able to imagine has amounted to something like, "But but but the world's just got to be made out of STUFF!!!" But I'm certainly not operating under the assumption that these are the best possible objections. I'd love to see what happens with whatever you've got to throw at my position.
The theory you propose in (2) seems close to Neutral Monism. It has fallen into disrepute (and near oblivion) but was the preferred solution to the mind-body problem of many significant philosophers of the late 19th-early 20th, in particular of Bertrand Russell (for a long period). A quote from Russell:
If I knew how the brain worked in sufficient detail, I think I'd be able to explain why this was wrong; I'd have a theory that would predict what qualia a brain experiences based on its structure (or whatever). No, I don't know what the theory is, but I'm pretty confident that there is one.
Can you give me an example of how, even in principle, this would work? Construct a toy universe in which there are experiences causally determined by non-experiences. How would examining anything about the non-experiences tell us that the experiences exist, or what particular way those experiences feel?
http://lesswrong.com/lw/p5/brain_breakthrough_its_made_of_neurons/
http://lesswrong.com/lw/p3/angry_atoms/
So your argument is "Doing arithmetic requires consciousness; and we can tell that something is doing arithmetic by looking at its hardware; so we can tell with certainty by looking at certain hardware states that the hardware is sentient"?
So your argument is "We have explained some things physically before, therefore we can explain consciousness physically"?
So your argument is "Mental states have physical causes, so they must be identical with certain brain-states"?
Set aside whether any of these would satisfy a dualist or agnostic; should they satisfy one?
Can you give me an example of how, even in principle, this would work? Construct a toy universe in which there are computations causally determined by non-computations. How would examining anything about the non-computations tell us that the computations exist, or what particular functions those computations are computing?
My initial response is that any physical interaction in which the state of one thing differentially tracks the states of another can be modeled as a computation. Is your suggestion that an analogous response would solve the Hard Problem, i.e., are you endorsing panpsychism ('everything is literally conscious')?
What you described as computation could apply to literally any two things in the same causal universe. But you meant two things that track each other much more tightly than usual. It may be that a rock is literally conscious, but if so, then not very much so. So little that it really does not matter at all. Humans are much more conscious because they reflect the world much more, reflect themselves much more, and [insert solution to Hard Problem here].
I dunno. I think if rocks are even a little bit conscious, that's pretty freaky, and I'd like to know about it. I'd certainly like to hear more about what they're conscious of. Are they happy? Can I alter them in some way that will maximize their experiential well-being? Given how many more rocks there are than humans, it could end up being the case that our moral algorithm is dominated by rearranging pebbles on the beach.
Hah. Luckily, true panpsychism dissolves the Hard Problem. You don't need to account for mind in terms of non-mind, because there isn't any non-mind to be found.
I meant, I'm pretty sure that rocks are not conscious. It's just that the best way I'm able to express what I mean by "consciousness" may end up apparently including rocks, without me really claiming that rocks are conscious like humans are - in the same way that your definition of computation literally includes air, but you're not really talking about air.
I don't understand this. How would saying "all is Mind" explain why qualia feel the way they do?
In general, I would suggest as much looking at sensory experiences that vary among humans; there's already enough interesting material there without wondering if there are even other differences. Can we explain enough interesting things about the difference between normal hearing and pitch perfect hearing without talking about qualia?
Once we've done that, are we still interested in discussing qualia in color?
Daniel Dennett's 'Quining Qualia' (http://ase.tufts.edu/cogstud/papers/quinqual.htm) is taken ('round these parts) to have laid the theory of qualia to rest. Among philosophers, the theory of qualia and the classical empiricism founded on it are also considered to be dead theories, though it's Sellers "Empiricism and the Philosophy of Mind" (http://www.ditext.com/sellars/epm.html) that is seen to have done the killing.
I'm not at all convinced that all LWers have been persuaded that they don't have qualia.
Amongst some philosophers.
Hmmm. The only enthusiast for Sellars I know finds it necessary to adopt Direct Realism, which is a horribly flawed theory. In fact most of the problems with it consist of reconciling it with a naturalistic world view.
Well, it's probably important to distinguish between to uses to which the theory of qualia is put: first as the foundation of foundationalist empiricism, and second as the basis for the 'hard problem of consciousness'. Foundationalist theories of empiricism are largely dead, as is the idea that qualia are a source of immediate, non-conceptual knowledge. That's the work that Sellars (a strident reductivist and naturalist) did.
Now that I read it again, I think my original post was a bit misleading because I implied that the theory of qualia as establishing the 'hard problem' is also a dead theory. This is not the case, and important philosophers still defend the hard problem on these grounds. Mea Culpa.
Once direct realism as an epistemic theory is properly distinguished from a psychological theory of perception, I think it becomes an extremely plausible view. I think I'd probably call myself a direct realist.
Right - to hammer on the point, the common-ish (EDIT: Looks like I was hastily generalizing) LW opinion is that there never was any "hard problem of consciousness" (EDIT: meaning one that is distinct from "easy" problems of consciousness, that is, the ones we know roughly how to go about solving). It's just that when we meet a problem that we're very ignorant about, a lot of people won't go "I'm very ignorant about this," they'll go "This has a mysterious substance, and so why would learning more change that inherent property?"
It should be remembered though that the guy who's famous for formulating the hard problem of consciousness is:
1) A fan of EY's TDT, who's made significant efforts to get the theory some academic attention. 2) A believer in the singularity, and its accompanying problems. 3) The student of Douglas Hofstrader. 4) Someone very interested in AI. 5) Someone very well versed and interested in physics and psychology. 6) A rare, but sometimes poster on LW. 7) Very likely one of the smartest people alive. etc. etc.
I think consciousness is reducible too, but David Chalmers is a serious dude, and the 'hard problem' is to be taken very, very seriously. It's very easy to not see a philosophical problem, and very easy to think that the problem must be solved by psychology somewhere, much harder to actually explain a solution/dissolution.
Though on the other hand, we don't have room to take everything serious dudes say seriously - too many dudes, not enough time.
If a problem happens not to exist, then I suppose one will just have to nerve onesself and not see it. Yes, there are non-hard problems of consciousness, where you explain how a certain process or feeling occurs in the brain, and sure, there are some non-hard problems I'd wave away with "well, that's solved by psychology somewhere." But no amount of that has any bearing on the "hard problem," which will remain in scare quotes as befits its effective nonexistence - finding a solution to a problem that is not a problem would be silly.
(EDIT: To clarify, I am not saying qualia do not exist, I am saying some mysterious barrier of hardness around qualia does not exist.)
For practical reasons, I think that's fair enough...so long as we're clear that the above is a fully general counterargument.
Right. I have not said any actual arguments against the hard problem of consciousness.
EDIT: Was true when I said it, then I replied to PeterD, not that it worked (as I noted in that very post, the direct approach has little chance against a confusion)
Argument for the importance of the HP: it is about the only thing that would motivate an educated 21st century person into doubting physcalism.
OK. Then demonstrate that the HP does not exist, in terms of Chalmer's specification, by showing that we do have a good explanation.
Well, said Achilles, everybody knows that if you have A and B and "A and B imply Z," then you have Z.
How an Algorithm Feels From Inside.
The Visual Cortex is Used to Imagine
Stimulating the Visual Cortex Makes the Blind See
This sort of thing is sufficient for me, like Achilles' explanations were enough for Achilles. But if, say, the perception of the hard problem was causally unrelated to the actual existence of a hard problem (for epiphenominalism, this is literally what is going on), then gosh, it would seem like no matter what explanations you heard, the hard problem wouldn't go away - so it must be either a proof of dualism or a mistake.
But not for me. Indeed. I am pretty sure none of those articles is even intended as a solution to the HP. And if they are, why not publish them is a journal and become famous?
Intended as a solution to FW.
So? Every living qualiaphile accepts some sort of relationship between brain states and qualia.
So? I said nothing about epiphenomenalism
The non-parenthetical was a throwback to a whole few posts ago, where I claimed that perception of the hard problem was often from the mind projection fallacy.
Other than that, I don't have much to respond to here, since you're just going "So?"
I can't find the posting, and I don't see how the MPF would relate to e12ism anyway.
How did you expect to convive me? I am familar with all the stuff you are quoting, and I still think there is an HP. So do many people.
I agree with you about how smart Chalmers is and that he does very good philosophical work. But I think you have a mistake in terminology when you say
It is an understandable mistake, because it is natural to take "the hard problem" as meaning just "understanding consciousness", and I agree that this is a hard problem in ordinary terms and that saying "there is a reduction/dissolution" is not enough. But Chalmers introduced the distinction between the "hard problem" and the "easy problems" by saying that understanding the functional aspects of the mind, the information processing, etc, are all "easy problems". So a functionalist/computationalist materialist, like most people on this site, cannot buy into the notion that there is a serious "hard problem" in Chalmers' sense. This notion is defined in a way that begs the question assuming that qualia are irreducible. We should say instead that solving the "easy problems" is at the same time much less trivial than Chalmers makes it seem, and enough to fully account for consciousness.
No it isn't. Here is what Chalmers says:
"It is undeniable that some organisms are subjects of experience. But the question of how it is that these systems are subjects of experience is perplexing. Why is it that when our cognitive systems engage in visual and auditory information-processing, we have visual or auditory experience: the quality of deep blue, the sensation of middle C? How can we explain why there is something it is like to entertain a mental image, or to experience an emotion? It is widely agreed that experience arises from a physical basis, but we have no good explanation of why and how it so arises. Why should physical processing give rise to a rich inner life at all? It seems objectively unreasonable that it should, and yet it does."
There is no statement of irreducubility there. There is a statement that we have "no good explanaion" and we don't.
However, see how he contrasts it with the "easy problems" (from Consciousness and its Place in Nature - pdf):
It seems clear that for Chalmers any description in terms of behavior and cognitive function is by definition not addressing the hard problem.
But that is not to say that qualia are irreducibole things, that is to say that mechanical explanations of qualia have not worked to date
What does this mean by "why"? What evolutionary advantage is there? Well, it enables imagination, which lets us survive a wider variety of dangers. What physical mechanism is there? That's an open problem in neurology, but they're making progress.
I've read this several times, and I don't see a hard philosophical problem.
It's definitely a how-it-happens "why" and not how-did-it-evolve "why"
There's more to qualia than free-floating representations. There is no reason to suppose an AI's internal maps have phenomenal feels, no way of testing that they do, and no way of engineering them in.
It's a hard scientific problem. How could you have a theory that tells you how the world seems to a bat on LSD? How can you write a SeeRed() function?
There is a Hard Prolem, becuase there is basically no (non eliminative) science or technology of qualia at all. We cna get a start on the problem of building cognition, memory and perception into an AI, but we can;t get a start on writing code for Red or Pain or Salty. You can thell there is basically no non-eliminative science or technology of qualia because the best LWers' can quote is Dennett's eliminative theory.
The rest mostly go, "this could only be explained by a mysterious substance, there are no mysterious substances, therefore this does not exist."
I don't know why you guys keep harping about substances. Substance dualism has been out of favour for a good century.
Sorry, I was misusing terminology. Any ignorance-generating / ignorance-embodying explanation (e.g.s quantum mysticism / elan vital) uses what I'm calling "mysterious substance."
Basically I'm calling "quantum" a mysterious substance (for the quantum mystics), even though it's not like you can bottle it.
Maybe I should have said "mysterious form?" :D
I haven't read either of those but will read them. Also I totally think there was a respectable hard problem and can only stare somewhat confused at people who don't realize what the fuss was about. I don't agree with what Chalmers tries to answer to his problem, but his attempt to pinpoint exactly what seems so confusing seems very spot-on. I haven't read anything very impressive yet from Dennett on the subject; could be that I'm reading the wrong things. Gary Drescher on the other hand is excellent.
It could be that I'm atypical for LW.
EDIT: Skimmed the Dennett one, didn't see much of anything relatively new there; the Sellers link fails.
I'll take a look at Drescher, I haven't seen that one.
Try this link? http://selfpace.uconn.edu/class/percep/SellarsEmpPhilMind.pdf
Sellars is important to contemporary philosophy, to the extent that a standard course in epistemology will often end with EPM. I'm not sure it's entirely worth your time though, because an argument against classical (not Bayesian) empiricism.
Pryor and BonJour explain Sellars better than Sellars does. See: http://www.jimpryor.net/teaching/courses/epist/notes/given.html
The basic question is over whether our beliefs are purely justified by other beliefs, or whether our (visual, auditory, etc.) perceptions themselves 'represent the world as being a certain way' (i.e., have 'propositional content') and, without being beliefs themselves, can lend some measure of support to our beliefs. Note that this is a question about representational content (intentionality) and epistemic justification, not about phenomenal content (qualia) and physicalism.
So you do have a solution to the problem?
Do you have evidence of this? The PhilPapers survey suggests that only 56.5% of philosophers identify as 'physicalists,' and 59% think that zombies are conceivable (though most of these think zombies are nevertheless impossible). It would also help if you explained what you mean by 'the theory of qualia.'
Sellars' argument, I think, rests on a few confusions and shaky assumptions. I agree this argument is still extremely widely cited, but I think that serious epistemologists no longer consider it conclusive, and a number reject it outright. Jim Pryor writes:
I mentioned in a subsequent post that there was an ambiguity in my original claim. Qualia have been used by philosophers to do two different jobs: 1) as the basis of the hard problem of consciousness, and 2) as the foundation of foundationalist theories of empiricism. Sellars essay, in particular is aimed at (2), not (1), and the mention of 'qualia' to which I was responding was probably a case of (1). The question of physicalism and the conceivability of p-zombies isn't directly related to the epistemic role of qualia, and one could reject classical empiricism on the basis of Sellars' argument while still believing that the reality of irreducible qualia speak against physicalism and for the conceivability of p-zombies.
That may be, it's a bit outside my ken. Thanks for posting the quote. I won't go trying to defend the overall organization EPM, which is fairly labyrinthine, but I have some confidence in its critiques: I'd need more familiarity with Pryor's work to level a serious criticism, but he on the basis of your quote he seems to me to be missing the point: Sellars is not arguing that something's appearing to you in a certain way is a state (like a belief) which requires justification. He argues that it is not tenable to think of this state as being independent of (e.g. a foundation for) a whole battery of concepts including epistemic concepts like 'being in standard perceptual conditions'. Looking a certain way is posterior (a sophistication of) its being that way. Looking red is posterior to simply being red. And this is an attack on the epistemic role of qualia insofar as this theory implies that 'looking red' is in some way fundamental and conceptually independent.
Yes, that is the argument. And I think its soundness is far from obvious, and that there's a lot of plausibility to the alternative view. The main problem is that this notion of 'conceptual content' is very hard to explicate; often it seems to be unfortunately confused with the idea of linguistic content; but do we really think that the only things that should add or take away any of my credence in any belief is the words I think to myself? In any case, Pryors' paper Is There Non-Inferential Justification? is probably the best starting point for the rival view. And he's an exceedingly lucid thinker.
I'll read the Pryor article, in more detail, but from your gloss and from a quick scan, I still don't see where Pryor and Sellars are even supposed to disagree. I think, without being totally sure, that Sellars would answer the title question of Pryor's article with an emphatic 'yes!'. Experience of a red car justifies belief that the car is red. While experience of a red car also presupposes a battery of other concepts (including epistemic concepts), these concepts are not related to the knowledge of the redness of the car as premises to a conclusion.
Here's a quote from EPM p148, which illustrates that the above is Sellars' view (italics mine). Note that in the following, Sellars is sketching the view he wants to attack:
So Sellars wants to argue that empiricism has no foundation because experience (as an epistemic success term) is not possible without knowledge of a bunch of other facts. But it does not follow from this that a) Sellars thinks knowledge derived from experience is inferential, or b) Sellars thinks non-inferential knowledge as such is a problem.
But that said, I haven't read enough of Pryor's paper(s) to understand his critiques. I'll take a look.
I just read 'Quining Qualia'. I do not see it as a solution to the hard problem of consciousness, at all. However, I did find it brilliant - it shifted my intuition from thinking that conscious experience is somehow magical and inexplicable to thinking that it is plausible that conscious experience could, one day, be explained physically. But to stop here would be to give a fake explanation...the problem has not yet been solved.
-- Eliezer Yudkowsky, Dissolving the Question
Also, does anyone disagree with anything that Dennett says in the paper, and, if so, what, and why?
I think I have qualia. I probably don't have qualia as defined by Dennett, as simultaneously ineffable, intrinsic, etc, but there are nonetheless ways things seem to me.
I don't know what others accept as a solution to the qualia problem, but I've found the explanations in "How an algorithm feels from the inside" quite on spot. For me, the old sequences have solved the qualia problem, and from what I see the new sequence presupposes the same.
Stimulating as always! I have a criticism to make of the use made of the term 'rigid designation'.
What philosophers of language ordinarily mean by calling a term a rigid designator is not that, considered purely syntactically, it intrinsically refers to anything. The property of being a rigid designator is something which can be possessed by an expression in use in a particular language-system. The distinction is between expressions-in-use whose reference we let vary across counterfactual scenarios (or 'possible worlds'), e.g. 'The first person to climb Everest', and those whose reference remains stable, e.g. 'George Washington', 'The sum of two and two'.
There is some controversy over how to apply the rigid/non-rigid distinction to general terms like 'fair' (or predicates like 'is fair') - cf. Scott Soames' book Beyond Rigidity - but I think the natural thing to say is that 'is fair' is rigid, since it is used to attribute the same property across counterfactual scenarios, in contrast with a predicate like 'possesses my favourite property'.
I just wanted to agree with Tristanhaze here that this usage strikes me as non-standard. I want to put this in my own words so that Tristanhaze/Eliezer/others can correct me if I've got the wrong end of the stick.
If something is a rigid designator it means that it refers to the same thing in all possible worlds. To say it's non-rigid is to say it refers to different things in some possible worlds to others. This has nothing to do with whether different language users that use the phrase must always be referring to the same thing. So George Washington may be a rigid designator in that it refers to the same person in all possible worlds (bracketing issues of transworld identity) but that doesn't mean that in all possible worlds that person is called George Washington or that in all possible worlds people who use the name George Washington must be referring to this person or even that in the actual world all people who use the name George Washington must be referring to this person.
To say "water" is a rigid designator is to say that whatever possible world I am talking about, I am picking out the same thing when I use the word water (in a way that I wouldn't be when I say, "the tallest person in the world" - this would pick out different things in different worlds). But it doesn't say anything about whether I mean the same thing as other language users in this or other possible worlds.
ETA: So the relevance to the quoted section is that rigid designators aren't about whether someone that thinks of Euclidean geometry when you say "numbers" is right or wrong - it's about whether whatever they associate with that word is the same thing in all possible worlds (or whether it's a different thing in some worlds).
ETA 2: I take it that Eliezer's paragraph here is in response to comments like these. I'm in a bit of a rush and need to think about it some more but I think Richard may be making a different point here to the one Eliezer's making (on my reading). I think Richard is saying that what is "right" is rigidly determined by my current (idealised) desires - so in a possible world where I desired to murder, murder would still be wrong because "right" is a rigid designator (that is, right from the perspective of my language, a different language user - like the me that desires murder - might still use "right" to refer to something else according to which murder is right. See the point about George Washington being able to be rigid even if people in other possible worlds use that name to name someone else). On the other hand, my reading of Eliezer was that he was taking the claim that "right" (or "fair") is a rigid designator to mean something about the way different language users use the word "fair". Eliezer seemed to be suggesting that rigid designation implied that words intrinsically mean certain things and hence that rigid designation implies that if someone uses a word in a different way they are wrong (using numbers to refer to geometry). I could have misunderstood either of these two comments but if I haven't then it seems to me that Eliezer is using rigid designator in a non-standard way.
Correct. Eliezer has misunderstood rigid designation here.
So does that mean this:
...is your real claim here, independent of any points about language use?
If so, I think I would just straightforwardly modify my paragraph above to say that my statements are not trying to talk about language use or human brains / desires, albeit that desire is both an optimization target of, and a quotation of, morality.
I'm not sure what you have in mind here. We need to distinguish (i) the referent of a concept from (ii) its reference-fixing "sense" or functional role. The way I understood your view, the reference-fixing story for moral terms involves our (idealized) desires. But the referent is "rigid" in the sense that it's picking out the content of our desires: the thing that actually fills the functional role, rather than the role-property itself.
Since our desires typically aren't themselves about our desires, so it will turn out, on this story, that morality is not "about" desires. It's about "love, friendship," and all that jazz. But there's a story to be told about how our moral concepts came to pick out these particular worldly properties. And that's where desires come in (as I understand your view). Our moral concepts pick out these particular properties because they're the contents of our idealized desires. But that's not to say that therefore morality is "really" just about fulfilling any old desires. For that would be to neglect the part that rigid designation, and the distinction between reference and reference-fixing, plays in this story.
Does that capture your view? To further clarify: the point of appealing to "rigid designation" is just to explain how desires could play a reference-fixing role without being any part of the referent of moral talk (or what it is "about"). Isn't that what you're after? Or do you have some other reference-fixing story in mind?
This all does sound good to me; but, is there a way to say the above while tabooing "reference" and avoiding talk of things "referring" to other things? Reference isn't ontologically basic, so what does it reduce to?
Basically, the main part that would worry me is a phrase like, "there's a story to be told about how our moral concepts came to pick out these particular worldly properties" which sounds on its face like, "There's a story to be told about how successorship came to pick out the natural numbers" whereas what I'd want to say is, "Of course, there's a story to be told about how moral concepts came to have the power to move us" or "There's a story to be told about how our brains came to reflect numbers".
I'd like to say "sure" and then delete that paragraph, but then somebody else in the comments will say that my essay is just talking about a rigid-designation theory of morality. I mean, that's the comment I've gotten multiple times previously. Anyone got a good idea for resolving this?
You may have resolved this now by talking to Richard (who knows more about this than me) but, in case you haven't, I'll have a shot at it.
First, the distinction: Richard is using rigid designation to talk about how a single person evaluates counterfactual scenarios, whereas you seem to be taking it as a comment about how different people use the same word.
Second, relevance: Richard's usage allow you to respond to an objection. The objection asks you to consider the counterfactual situation where you desire to murder people and says murder must now be right so the theory is extremely subjective. You can respond that "right" is a rigid designator so it is still right to not murder in this counterfactual situation (though your counterpart here will use the word "right" differently).
Suggestion: perhaps edit the paragraph so as to discuss either this objection and defence or outline why the rigid designator view so characterised is not your view.
I am having difficulty understanding the model of 'physics+logic = reality.' Up until now I have understood that's physics was reality, but logic is the way to describe and think about what follows from it. Would someone please post a link to the original article (in this sequence or not) which explains the position? Thank you.
Thank you.
I read this post with a growing sense of unease. The pie example appears to treat "fair" as a 1-place word, but I don't see any reason to suppose it would be. (I note my disquiet that we are both linking to that article; and my worry about how confused this post seems to me.)
The standard atheist reply is tremendously unsatisfying; it appeals to intuition and assumes what it's trying to prove!
My resolution of Euthryphro is "the moral is the practical." A predictable consequence of evolution is that people have moral intuitions, that those intuitions reflect their ancestral environment, and that those intuitions can be variable. Where would I find mercy, justice, or duty? Cognitive algorithms and concepts inside minds.
This article reads like you're trying to move your stone tablet from your head into the world of logic, where it can be as universal as the concept of primes. It's not clear to me why you're embarking on that particular project.
The example of elegance seems like it points the other way. If your sense of elegance is admittedly subjective, why are we supposing a Platonic form of elegance out in the world of logic? Isn't this basically the error where one takes a cognitive algorithm that recognizes whether or not something is a horse and turns it into a Platonic form of horseness floating in the world of logic?
It looks to me like you're trying to say "because classification algorithms can be implemented in reality, there can be real ensembles that embody logical facts, but changing the classification algorithms doesn't change those logical facts," which seems true but I don't see what work you expect it to do.
There's also the statement "when you change the algorithms that lead to outputs, you change the internal sensation of those outputs." That has not been my experience, and I don't see a reason why that would be the case. In particular, when dreaming it seems like many algorithms have their outputs fixed at certain values: my 'is this exciting?' algorithm may return 'exciting!' during the dream but 'boring!' when considering the dream whilst awake, but the sensation that results from the output of the algorithm seems indistinguishable; that is, being excited in a dream feels the same to me as being excited while awake. (Of course, it could be that whichever part of me is able to differentiate between sensations is also malfunctioning while dreaming!)
If you show me the pattern of neurons firing that happens when my bladder is full, then my bladder won't feel full. If you put an electrode in my head (or use induction, or whatever) and replicate that pattern of neurons firing, then my bladder will feel full, because the feeling of fullness is the output of those neurons firing in that pattern.
You sure it's not just executing an adaption? Why?
How do you avoid prudent predation
Are there moral systems used by humans that avoid prudent predation, and are not outcompeted by moral systems used by humans that make use of prudent predation?
I will note that the type of predation that is prudent has varied significantly over time, and correspondingly, so have moral intuitions. Further altering the structure of society will again alter the sort of predation that is prudent, and so one can seek to restructure society so disliked behavior is less prudent and liked behavior is more prudent.
I find it hard to make sense of that. I don't think people go in for morality for selfish gain, and the very idea may be incoherent.
Maybe. I don't see what your point is. If the moral is not the practical, and if PP is wrong, that would not imply morality is timeless, and vice versa.
The claim is that moral intuitions exist because they were selected for, and they must have been selected for because they increased reproductive fitness. Similarly, we should expect moral behavior to the degree that morality is more rewarding than immorality. (The picture is muddied by there being both genetic and memetic evolution, but the basic idea survives.)
But morality isn't just moral intuitions. It includes "eat fish on friday"
That doens't follow. Fitness-enahncing and gene-spreading behaviour don;t have to reward the organism concerned. What't the reward for self sacrifice?
that's a considerable understatement.
Sure. We should expect such rules to be followed to the degree that they are prudent.
There are several; kin selection, reciprocal altruism, and so on. In some cases, self-sacrifice is the result of a parasitic relationship. (Kin selection appears to have a memetic analog as well, but I'm not familiar with work that develops that concept rigorously, and distinguishes it from normal alliance behaviors; it might just be a subset of that.)
Again, I have no idea what you mean. Morality does not predict self-centered prudence, since it enjoins self-sacrifice, and evolution doenst predict self-centered prudence in all cases. It is not selfishly prudent for bees to to defend their colony, or for male praying mantises to mate.
Rewards for whom?
I think the author of that piece needs to learn the concept of precommitment. Precommitting to one-box is not at all the same as believing that one-boxing is the dominant strategy in the general newcomb problem. Likewise, precommitting not to engage in prudent predation is not a matter of holding a counterfactual belief, but of taking a positive-expected-utility action.
It is exactly executing an adaption. No "just" about it though. An AI programmed to maximise paperclips is motivated by increasing the number of paperclips. It's executing its program.
I had this post in mind. I see no reason to link behavior that 'seems moral' to the internal sensation of motivation by those terminal values, and if we're not talking about introspection about decision-making, then why are we using the word motivation?
This post seems to be discussing a particular brand of moral reasoning- basically, deliberative utilitarian judgments- which seems like a rather incomplete picture of human morality as a whole, and it seems like it's just sweeping under the rug the problem of where values come from in the first place. I should make clear that first he has to describe what values are before he can describe where values come from, but if it's an incomplete description of values, that can cause problems down the line.
'Beautiful' needs 2 places because our concept of beauty admits of perceptual variation. 'Fairness' does not grammatically need an 'according to whom?' argument place, because our concept of fairness is not observer-relative. You could introduce a function that takes in a person X who associates a definition with 'fairness,' takes in a situation Y, and asks whether X would call Y 'fair.' But this would be a function for 'What does the spoken syllable FAIR denote in a linguistic community?', not a function for 'What is fair?' If we applied this demand generally, 'beautiful' would became 3-place ('what objects X would some agent Y say some agent Z finds 'beautiful'?'), as would logical terms like 'plus' ('how would some agent X perform the operation X calls "addition" on values Y and Z?'), and indeed all linguistic acts.
Yes, but a given intuition cannot vary limitlessly, because there are limits to what we would consider to fall under the same idea of 'fairness.' Different people may use the spoken syllables FAIR, PLUS, or BEAUTIFUL differently, but past a certain point we rightly intuit that the intension of the words, and not just their extension, has radically changed. Thus even if 'fairness' is disjunctive across several equally good concepts of fairness, there are semantic rules for what gets to be in the club. Plausibly, 'fairness is whatever makes RobbBB happiest' is not a semantic candidate for what English-speakers are logically pinpointing as 'fairness.'
You hear 'Oh no, he's making morality just as objective as number theory!' whereas I hear 'Oh good, he's making morality just as subjective as number theory.' If we can logically pinpoint 'fairness,' then fairness can be rigorously and objectively discussed even if some species find the concept loathsome; just as if we can logically pinpoint 'prime number,' we can rigorously and objectively discuss the primes even with a species S who finds it unnatural to group 2 with the other primes, and a second species S* who finds it unnatural to exclude 1 from their version of the primes. Our choice of whether to consider 2 prime, like our choice of which semantic value to assign to 'fair,' is both arbitrary and unimpeachably objective.
Or do you think that number theory is literally writ into the fabric of reality somewhere, that Plato's Heaven is actually out there and that we therefore have to be very careful about which logical constructs we allow into the club? This reluctance to let fairness into an elite Abstraction Club, even if some moral codes are just as definable in logical terms as is number theory, reminds me of Plato's neurotic reluctance (in the Parmenides) to allow for the possibility that there might be Forms "of hair, mud, dirt, or anything else which is vile and paltry." Constructible is constructible; there is not a privileged set of Real Constructs distinct from the Mere Fictions, and the truths about Sherlock Holmes, if defined carefully enough, get the same epistemic and metaphysical status as the truths about Graham's Number.
You're confusing epistemic subjectivity with ontological subjectivity. Terms that are defined via or refer to mind- or brain-states may nevertheless be defined with so much rigor that they admit no indeterminacy, i.e., an algorithm could take in the rules for certain sentences about subjectivity and output exactly which cases render those sentences true, and which render them false.
What makes you think that the 'world of logic' is Platonic in the first place? If logic is a matter of mental construction, not a matter of us looking into our metaphysical crystal balls and glimpsing an otherworldly domain of Magical Nonspatiotemporal Thingies, then we cease to be tempted by Forms of Horsehood for the same reason we cease to be tempted by Forms of Integerhood.
What? It seems to me that fairness and beauty are equally subjective, and the intuition that says "but my sense of fairness is objectively correct!" is the same intuition that says "but my sense of beauty is objectively correct!"
I agree that we can logically pinpoint any specific concept; to use the pie example, Yancy uses the concept of "splitting windfalls equally by weight" and Zaire uses the concept of "splitting windfalls equally by desire." What I disagree with is the proposition that there is this well-defined and objective concept of "fair" that, in the given situation, points to "splitting windfalls equally by weight."
One could put forward the axiom that "splitting windfalls equally by weight is fair", just like one can put forward the axiom that "zero is not the successor of any number," but we are no closer to that axiom having any decision-making weight; it is just a model, and for it to be used it needs to be a useful and appropriate model.
"Fair", quoted, is a word. You don't think it's plausible that in English "fair" could refer to splitting windfalls equally by weight? (Or rather to something a bit more complicated that comes out to splitting windfalls equally by weight in the situation of the three travellers and the pie.)
I agree that someone could mean "splitting windfalls equally by weight" when they say "fair." I further submit that words can be ambiguous, and someone else could mean "splitting windfalls equally by desire" when they say "fair." In such a case, where the word seems to adding more heat than light, I would scrap it and go with the more precise phrases.
I don't know what you mean by 'subjective.' But perhaps there is a (completely non-denoting) concept of Objective Beauty in addition to the Subjective Beauty ('in the eye of the beholder') I'm discussing, and we're talking past each other about the two. So let's pick a simpler example.
'Delicious' is clearly two-place, and ordinary English-language speakers routinely consider it two-place; we sometimes elide the 'delicious for whom?' by assuming 'for ordinary humans,' but it would be controversial to claim that speaking of deliciousness automatically commits you to a context-independent property of Intrinsic Objective Tastiness.
Now, it sounded like you were claiming that fairness is subjective in much the same way as deliciousness; no claim about fairness is saturated unless it includes an argument place for the evaluater. But this seems to be false simply given how people conceive of 'fair' and 'delicious'. People don't think there's an implicit 'fairness-relative-to-a-judge-thereof' when we speak of 'fairness,' or at least it don't think it in the transparent way they think of 'deliciousness' as always being 'deliciousness-relative-to-a-taster.' ('Beauty,' perhaps, is an ambiguous case straddling these two categories.) So is there some different sense in which fairness is 'subjective'? What is this other sense?
Are you claiming that Eliezer lacks any well-defined concept he's calling 'fairness'? Or are you claiming that most English-speakers don't have Eliezer's well-defined fairness in mind when they themselves use the word 'fair,' thus making Eliezer guilty of equivocation?
People argue about how best to define a term all the time, but we don't generally conclude from this that any reasoning one proceeds to carry out once one has stipulated a definition for the controversial term is for that reason alone 'subjective.' There have been a number of controversies in the history of mathematics — places where people's intuitions simply could not be reconciled by any substantive argument or proof — and mathematicians responded by stipulating precisely what they meant by their terms, then continuing on from there. Are you suggesting that this same method stops being useful or respectable if we switch domains from reasoning about this thing we call 'quantity' to reasoning about this thing we call 'fairness'?
What would it mean for an axiom to have "decision-making weight"? And do you think Eliezer, or any other intellectually serious moral realist, is honestly trying to attain this "decision-making weight" property?
That the judgments of "fair" or "beautiful" don't come from a universal source, but from a particular entity. I have copious evidence that what I consider "beautiful" is different from what some other people consider "beautiful;" I have copious evidence that what I consider "fair" is different from what some other people consider "fair."
It is clear to me that delicious is two-place, but it seems to me that people have to learn that it is two-place, and evidence that it is two-place is often surprising and potentially disgusting. Someone who has not learned through proverbs and experience that "beauty is in the eye of the beholder" and "there's no accounting for taste" would expect that everyone thinks the same things are beautiful and tasty.
There are several asymmetries between them. Deliciousness generally affects one person, and knowing that it varies allows specialization and gains from trade (my apple for your banana!). Fairness generally requires at least two people to be involved, and acknowledging that your concept of fairness does not bind the other person puts you at a disadvantage. Compare Xannon's compromise to Yancy's hardlining.
People thinking that something is objective is not evidence that it is actually objective. Indeed, we have plenty of counterevidence in all the times that people argue over what is fair.
No? I'm arguing that Eliezer::Fair may be well-defined, but that he has put forward no reason that will convince Zaire that Zaire::Fair should become Eliezer::Fair, just like he has put forward no reason why Zaire::Favorite Color should become Eliezer::Favorite Color.
There are lots of possible geometries out there, and mathematicians can productively discuss any set of non-contradictory axioms. But only a narrow subset of those geometries correspond well with the universe that we actually live in; physicists put serious effort into understanding those, and the rest are curiosities.
(I think that also answers your last two questions, but if it doesn't I'll try to elaborate.)
I don't understand what this means. To my knowledge, the only things that exist are particulars.
I have copious evidence that others disagree with me about ¬¬P being equivalent to P. And I have copious evidence that others disagree with me about the Earth's being more than 6,000 years old. Does this imply that my belief in Double Negation Elimination and in the Earth's antiquity is 'subjective'? If not, then what extra premises are you suppressing?
Well, sure. But, barring innate knowledge, people have to learn everything at some point. 3-year-olds lack a theory of mind; and those with a new theory of mind may not yet understand that 'beautiful' and 'delicious' are observer-relative. But that on its own gives us no way to conclude that 'fairness' is observer-relative. After all, not everything that we start off thinking is 'objective' later turns out to be 'subjective.'
And even if 'fairness' were observer-relative, there have to be constraints on what can qualify as 'fairness.' Fairness is not equivalent to 'whatever anyone decides to use the word "fairness" to mean,' as Eliezer rightly pointed out. Even relativists don't tend to think that 'purple toaster' and 'equitable distribution of resources' are equally legitimate and plausible semantic candidates for the word 'fairness.'
That's not true. Deliciousness, like fairness, affects everyone. For instance, my roommate is affected by which foods I find delicious; it changes where she ends up going to eat.
Perhaps you meant something else. You'll have to be much more precise. The entire game when it comes to as tricky a dichotomy as 'objective/subjective' is just: Be precise. The dichotomy will reveal its secrets and deceptions only if we taboo our way into its heart.
What's fair varies from person to person too, because different people, for instance, put different amounts of work into their activities. And knowing about what's fair can certainly help in trade!
Does not "bind" the other person? Fairness is not a physical object; it cannot bind people's limbs. If you mean something else by 'bind,' please be more explicit.
What would it mean for Zaire::Fair to become Eliezer::Fair? Are you saying that Eliezer's fairness is 'subjective' because he can't give a deductive argument from the empty set of assumptions proving that Zaire should redefine his word 'fair' to mean what Eliezer means by 'fair'? Or are you saying that Eliezer's fairness is 'subjective' because he can't give a deductive argument from the empty set of assumptions proving that Zaire should pretend that Zaire's semantic value for the word 'fair' is the same as Eliezer's semantic value for the word 'fair'? Or what? By any of these standards, there are no objective truths; all truths rely on fixing a semantic value for your linguistic atoms, and no argument can be given for any particular fixation.
They can also productively discuss sets of contradictory axioms, especially if their logic be paraconsistent.
So, since we don't live in Euclidean space, Euclidean geometry is merely a 'curiosity.' Is it, then, subjective? If not, what ingredient, what elemental objectivium, distinguishes Euclidean geometry from Yudkowskian fairness?
Your choice of logical system and your belief in an old Earth reside in your mind, and that you believe them only provides me rather weak evidence that they are beliefs I should hold. (I do hold both of those beliefs, but because of other evidence.) It is not clear to me that attaching the label of "subjective" or "objective" would materially improve my description.
When I write the word "generally," I mean it as a qualifier that acknowledges many objections could be raised that do not materially alter the point. Generally, at restaurants, you and your roommate are not required to eat the same meal, and the effects of, say, the unpleasant-to-her smell of your meal are smaller than the effects of the pleasant-to-her taste of her meal. Of course there are meals you could eat and restaurants you could choose where that is not case, but the asymmetry between the impact of your tastes on your roommate and the impact of your sense of fairness on your roommate remains in the general case.
Consider:
The priest walks by the beggar without looking. The beggar calls up to him, "Matthew 25!" Matthew is not the priest's name, but still he stops, decides, and then gives the beggar his sack lunch.
The practical use of moral and ethical systems is as a guide to decision-making. Moral systems typically specialize in guiding decisions in a way that increases the positive benefit to others and decreases the negative cost to them. Moral and ethical systems are only relevant insofar as they are used to make decisions.
The space we live in corresponds well (obviously, not perfectly) with Euclidean space, and so it receives significant attention from physicists. The space we live in doesn't correspond well with the Poincare disk hyperbolic geometry; the most likely place a non-mathematician has seen it is M.C. Escher.
Models of Euclidean geometry exist in minds, and one person's model of it may not agree with another's, but there is currently an established definition (i.e. blueprint for a model), and not using that definition correctly makes conversation about that topic difficult with humans who use the established definition. Comparing it to beauty, models of beauty exist in minds, and those models can differ, but there is not an established blueprint to construct a model of beauty.
But there is little upshot to people having differnt notions of beauty, since people can arrange their own environents to suit their own aesthetics. However, resources have to be apportioned one way or another. So we need, and have discussion about how to do things fairly. (Public architecture is a bit of an exception to what I said about beauty, but lo and behold, we have debates at that too).
Well, I'm glad to see you're taking a second crack at an exposition of metaethics.
I wonder if it might be worth expounding more on the distinction between utterances (sentences and word-symbols), meaning-bearers (propositions and predicates) and languages (which map utterances to meaning-bearers). My limited experience seems to suggest that a lot of the confusion about metaethics comes from not getting, instinctively, that speakers use their actual language, and that a sentence like "X is better than Y", when uttered by a particular person, refers to some fixed proposition about X and Y that doesn't talk about the definition of the symbols "X", "Y" and "better" in the speaker's language (and for that matter doesn't talk about the definitions of "is" and "than").¹
But I don't really know. I find it hard to get into people's heads in this case.
¹ In general. It is of course, possible that in some speaker's language "X" refers to something like the english language and "Y" refers to french, or that "better" refers to having more words for snow. But in general most things we say are not about language.
Um, how do you know?
It would depend on what exactly what we reprogrammed within you, I expect.
Exactly. I mean, you could probably make it have its own quale, but you could also make it not, and I don't see why that would be in question as long as we're postulating brain-reprogramming powers.
Assume the subject of reprogramming is an existing human being, otherwise minimally altered by this reprogramming, i.e., we don't do anything that isn't necessary to switch their motivation to paperclips. So unless you do something gratuitiously non-minimal like moving the whole decision-action system out of the range of introspective modeling, or cutting way down on the detail level of introspective modeling, or changing the empathic architecture for modeling hypothetical selves, the new person will experience themselves as having ineffable 'qualia' associated with the motivation to produce paperclips.
The only way to make it seem to them like their motivational quales hadn't changed over time would be to mess with the encoding of their previous memories of motivation, presumably in a structure-destroying way since the stored data and their introspectively exposed surfaces will not be naturally isomorphic. If you carry out the change to paperclip-motivation in the obvious way, cognitive comparisions of the retrieved memories to current thoughts will return 'unequal ineffable quales', and if the memories are visualized in different modalities from current thoughts, 'incomparable ineffable quales'.
Doing-what-leads-to-paperclips will also be a much simpler 'quale', both from the outside perspective looking at the complexity of cognitive data, and in terms of the internal experience of complexity - unless you pack an awful lot of detail into the question of what constitutes a more preferred paperclip. Otherwise, compared to the old days when you thought about justice and fairness, introspection will show that less questioning and uncertainty is involved, and that there are fewer points of variation among the motivational thought-quales being considered.
I suppose you could put in some extra work to make the previous motivations map in cognitively comparable ways along as many joints as possible, and try to edit previous memories without destroying their structure so that they can be visualized in a least common modality with current experiences. But even if you did, memories of the previous quales for rightness-motivation would appear as different in retrospect when compared to current quales for paperclip-motivation as a memory of a 3D greyscale forest landscape vs. a current experience of a 2D red-and-green fractal, even if they're both articulated in the visual sensory modality and your modal workspace allows you to search for, focus on, and compare commonly 'experienced' shapes between them.
This comment expands how you'd go about reprogramming someone in this way with another layer of granularity, which is certainly interesting on its own merits, but it doesn't strongly support your assertion about what it would feel like to be that someone. What makes you think this is how qualia work? Have you been performing sinister experiments in your basement? Do you have magic counterfactual-luminosity-powers?
It sounds to me like you don't think the answer had anything to do with the question. But to think that, you'd pretty much have to discard both the functionalist and physicalist theories of mind, and go full dualist/neutral monist; wouldn't you?
I think I'll go with this as my reply - "Well, imagine that you lived in a monist universe - things would pretty much have to work that way, wouldn't they?"
Possibly (this is total speculation) Eliezer is talking about the feeling of one's entire motivational system (or some large part of it), while you're talking about the feeling of some much narrower system that you identify as computing morality; so his conception of a Clippified human wouldn't share your terminal-ish drives to eat tasty food, be near friends, etc., and the qualia that correspond to wanting those things.
The Clippified human categorizes foods into a similar metric of similarity - still believes that fish tastes more like steak than like chocolate - but of course is not motivated to eat except insofar as staying alive helps to make more paperclips. They have taste, but not tastiness. Actually that might make a surprisingly good metaphor for a lot of the difficulty that some people have with comprehending how Clippy can understand your pain and not care - maybe I'll try it on the other end of that Facebook conversation.
The metaphor seems like it could lose most of its effectiveness on people who have never applied the outside view to how taste and tastiness feel from inside - they've never realized that chocolate tastes good because their brain fires "good taste" when it perceives the experience "chocolate taste". The obvious resulting cognitive dissonance (from "tastes bad for others") predictions match my observations, so I suspect this would be common among non-rationalists. If the Facebook conversation you mention is with people who haven't crossed that inferential gap yet, it might prove not that useful.
I think Eliezer is simply suggesting that qualia don't in fact exist in a vacuum. Green feels the way it does partly because it's the color of chlorophyll. In a universe where plants had picked a different color for chlorophyll (melanophyll, say), with everything else (per impossibile) held constant, we would associate an at least slightly different quale with green and with black, because part of how colors feel is that they subtly remind us of the things that are most often colored that way. Similarly, part of how 'goodness' feels is that it imperceptibly reminds us of the extension of good; if that extension were dramatically different, then the feeling would (barring any radical redesigns of how associative thought works) be different too. In a universe where the smallest birds were ten feet tall, thinking about 'birdiness' would involve a different quale for the same reason.
I think you and Alicorn may be talking past each other somewhat.
Throughout my life, it seems that what I morally value has varied more than what rightness feels like - just as it seems that what I consider status-raising has changed more than what rising in status feels like, and what I find physically pleasurable has changed more than what physical pleasures feel like. It's possible that the things my whole person is optimizing for have not changed at all, that my subjective feelings are a direct reflection of this, and that my evaluation of a change of content is merely a change in my causal model of the production of the desiderata (I thought voting for Smith would lower unemployment, but now I think voting for Jones would, etc.) But it seems more plausible to me that
1) the whole me is optimizing for various things, and these things change over time,
2) and that the conscious me is getting information inputs which it can group together by family resemblance, and which can reinforce or disincentivize its behavior.
Imagine a ship which is governed by an anarchic assembly beneath board and captained by an employee of theirs whom they motivate through in-kind bonuses. So the assembly at one moment might be looking for buried treasure, which they think is in such-and-such a place, and so they send her baskets of fresh apples when she's steering in that direction and baskets of stinky rotten apples when she's steering in the wrong. For other goals (refueling, not crashing into reefs) they send her excellent or tedious movies and gorgeous or ugly cabana boys. The captain doesn't even have direct access to what the apples or whatever are motivating her to do; although she can piece it together. She might even start thinking of apples as irreducibly connected to treasure. But if the assembly decided that they wanted to look for ports of call instead of treasure, I don't see why in principle they couldn't start sending her apples in order to do so. And if they did, I think her first response would be, if she was verbally asked, that the treasure - or whatever the dubloons constituting the treasure ultimately represent in terms of the desiderata of the assembly - had moved to the ports of call. This might be a correct inference - perhaps the assembly wants the treasure for money and now they think that comes better from heading to ports of call - but it hardly seems to be a necessarily correct one.
If I met two vampires, and one said his desire to drink blood was mediated through hunger (and that he no longer felt hunger for food, or lust) and another said her desire to drink blood was mediated through lust (and that she no longer felt lust for sex, or hunger) then I do think - presuming they were both once human, experiencing lust and hunger like me - they've told me something that allows me to distinguish their experiences from one another, even though they both desire blood and not food or sex.
They may or may not be able to explain to what it is like to be a bat.
Unless I'm inserting a further layer of misunderstanding your position seems to be curiously disjunctivist. I or you or Alicorn or all of us may be making bad inferences in taking "feels like" to mean "reminds one of the sort of experience that brings to mind..." ("I feel like I got mauled by a bear," says someone not just and maybe never mauled by a bear) or "constituting an experience of" ("what an algorithm feels like from the inside") when the other is intended. This seems to be a pretty easy elision to make - consider all the philosophers who say things like "well, it feels like we have libertarian free will..."
Wouldn't it be easier to have the programee remember themself as misunderstanding morality - like a reformed racist who previously preferred options that harmed minorities. I know when I gain more insight into my ethics I remember making decisions that, in retrospect, are incomprehensible (unless I deliberately keep in mind how I thought I should act.)
That depends on the details of how the human brain stores goals and memories.
Cached thoughts regularly supersede actual moral thinking, like all forms of thinking, and I am capable of remembering this experience. Am I misunderstanding your comment?
I wouldn't be all that suprised if the easiest way to get a human maximizing papperclips was to make it believe paperclips had epiphenomenal consciousnesses experiencing astronomical amounts of pleasure.
edit: or you could just give them a false memory of god telling them to do it.
Consider Bob. Bob, like most unreflective people, settles many moral questions by "am I disgusted by it?" Bob is disgusted by, among other things, feces, rotten fruit, corpses, maggots, and men kissing men. Internally, it feels to Bob like the disgust he feels at one of those stimuli is the same as the disgust he feels at the other stimuli, and brain scans show that they all activate the insula in basically the same way.
Bob goes through aversion therapy (or some other method) and eventually his insula no longer activates when he sees men kissing men.
When Bob remembers his previous reaction to that stimuli, I imagine he would remember being disgusted, but not be disgusted when he remembers the stimuli. His positions on, say, same-sex marriage or the acceptability of gay relationships have changed, and he is aware that they have changed.
Do you think this example agrees with your account? If/where it disagrees, why do you prefer your account?
I think this is really a sorites problem. If you change what's delicious only slightly, then deliciousness itself seems to be unaltered. But if you change it radically — say, if circuits similar to your old gustatory ones now trigger when and only when you see a bright light — then it seems plausible that the experience itself will be at least somewhat changed, because 'how things feel' is affected by our whole web of perceptual and conceptual associations. There isn't necessarily any sharp line where a change in deliciousness itself suddenly becomes perceptible; but it's nevertheless the case that the overall extension of 'delicious' (like 'disgusting' and 'moral') has some effect on how we experience deliciousness. E.g., deliciousness feels more foodish than lightish.
When I look at the problem introspectively, I can see that as a sensible guess. It doesn't seem like a sensible guess when I look at it from a neurological perspective. If the activation of the insula is disgust, then the claim that outputs of the insula will have a different introspective flavor when you rewire the inputs of the insula seems doubtful. Sure, it could be the case, but why?
When we hypnotize people to make them disgusted by benign things, I haven't seen any mention that the disgust has a different introspective flavor, and people seem to reason about that disgust in the exact same way that they reason about the disgust they had before.
This seems like the claim that rewiring yourself leads to something like synesthesia, and that just seems like an odd and unsupported claim to me.
Certain patterns of behavior at the insula correlate with disgust. But we don't know whether they're sufficient for disgust, nor do we know which modifications within or outside of the insula change the conscious character of disgust. There are lots of problems with identity claims at this stage, so I'll just raise one: For all we know, activation patterns in a given brain region correlate with disgust because disgust is experienced when that brain region inhibits another part of the brain; an experience could consist, in context, in the absence of a certain kind of brain activity.
Hypnosis data is especially difficult to evaluate, because it isn't clear (a) how reliable people's self-reports about introspection are while under hypnosis; nor (b) how reliable people's memories-of-hypnosis are afterward. Some 'dissociative' people even give contradictory phenomenological reports while under hypnosis.
That said, if you know of any studies suggesting that the disgust doesn't have at all a different character, I'd be very interested to see them!
If you think my claim isn't modest and fairly obvious, then it might be that you aren't understanding my claim. Redness feels at least a little bit bloodish. Greenness feels at least a little bit foresty. If we made a clone who sees evergreen forests as everred and blood as green, then their experience of greenness and redness would be partly the same, but it wouldn't be completely the same, because that overtone of bloodiness would remain in the background of a variety of green experiences, and that woodsy overtone would remain in the background of a variety of red experiences.
I'm differentiating between "red evokes blood" and "red feels bloody," because those seem like different things to me. The former deals with memory and association, and the second deals with introspection, and so I agree that the same introspective sensation could evoke very different memories.
The dynamics of introspective sensations could plausibly vary between people, and so I'm reluctant to discuss it extensively except in the context of object-level comparisons.
I'm not sure exactly what you mean by "red evokes blood." I agree that "red feels bloody" is intuitively distinct from "I tend to think explicitly about blood when I start thinking about redness," though the two are causally related. Certain shades of green to me feel fresh, clean, 'naturey;' certain shades of red to me feel violent, hot, glaring; certain shades of blue feel cool; etc. My suggestion is that these qualia, which are part of the feeling of the colors themselves for most humans, would be experientially different even when decontextualized if we'd gone through life perceiving forests as blue, oceans as red, campfires as green, etc. By analogy, the feeling of 'virtue' may be partly independent of which things we think of under the concept 'virtuous;' but it isn't completely independent of those things.
I think he's talking about the obvious fact that you'd be able to think to yourself "it seems I'm trying to maximize paperclips", as well as the other differences in your experience that would occur for similar reasons.
So far as I can tell, he chose to carve the world at this joint when making the definition of 'right'. In short, by definition. This is hardly the first time. Not too long ago, and perhaps in this sequence, there was a post about rightness and multiple-place-functions that justified the utility of this definition.
Speaking from personal experience, I can say that he's right.
Explaining how I know this, much less sharing the experience, is more difficult.
The simplest idea I can present is that you probably have multiple utility functions. If you're buying apples, you'll evaluate whether you like that type of apple, what the quality of the apple is, and how good the price is. For me, at least, these all FEEL different - a bruised apple doesn't "feel" overpriced the way a $5 apple at the airport does. Even disliking soft apples feels very different from recognizing a bruised apple, even though they both also go in to a larger basket of "no good".
What's more, I can pick apples based on someone ELSE'S utility function, and actually often shop with my roommate's function in mind (she likes apples a lot more than me, but is also much pickier, as it happens). This feels different from using my own utility function.
The other side of this is that I would expect my brain to NOTICE it's actual goals. If my goal is to make paperclips, I will think "I should do this because it makes paperclips", instead of "I should do this because it makes people happy". My brain doesn't have a generic "I should do this" emotion, as near as I can tell - it just has ways of signalling that an activity will accomplish my goals.
Thus, it seems reasonable to conclude that my feelings are more a combination of activity + outcome, not some raw platonic ideal. While sex, hiking, and a nice meal all make me "happy", they still feel completely different - I just lump them in to a larger category of "happiness" for some reason.
I'd strongly suspect you can add make-more-paperclips to that emotional category, but I see absolutely no reason you could make me treat it the same as a nice dinner, because that wouldn't even make sense.
Is this a fair summary?
The standard religious reply to the baby-slaughter dilemma goes something like this:
But that's just choosing the other horn of the dilemma, no? I.e., "god commands thing because they are moral."
And of course the atheist response to that is,
Not that anyone here didn't already know this, of course.
The wikipedia page lists some theistic responses that purport to evade both horns, but I don't recall being convinced that they were even coherent when I last looked at it.
It does choose a horn, but it's the other one, "things are moral because G-d commands them". It just denies the connotation that there exists a possible Counterfactual!G-d which could decide that Real!evil things are Counterfactual!good; in all possible worlds, G-d either wants the same thing or is something different mistakenly called "G-d". (Yeah, there's a possible world where we're ruled by an entity who pretends to be G-d and so we believe that we should kill babies. And there's a possible world where you're hallucinating this conversation.)
Or you could say it claims equivalence. Is this road sign a triangle because it has three sides, or does it have three sides because it is a triangle? If you pick the latter, does that mean that if triangles had four sides, the sign would change shape to have four sides? If you pick the former, does that mean that I can have three sides without being a triangle? (I don't think this one is quite fair, because we can imagine a powerful creator who wants immoral things.)
Three possible responses to the atheist response:
Sure. Not believing has bad consequences - you're wrong as a matter of fact, you don't get special believer rewards, you make G-d sad - but being immoral isn't necessarily one.
Well, you can be moral about most things, but worshiping my deity of choice is part of morality, so you can't be completely moral.
You could in theory, but how would you discover morality? Humans know what is moral because G-d told us (mostly in so many words, but also by hardwiring some intuitions). You can base your morality on philosophical reasoning, but your philosophy comes from social attitudes, which come from religious morality. Deviations introduced in the process are errors. All you're doing is scratching off the "made in Heaven" label from your ethics.
Obvious further atheist reply to the denial of counterfactuals: If God's desires don't vary across possible worlds there exists a logical abstraction which only describes the structure of the desires and doesn't make mention of God, just like if multiplication-of-apples doesn't vary across possible worlds, we can strip out the apples and talk about the multiplication.
How would a theist (at least the somewhat smart theist I'm emulating) disagree with that? That sounds a lot like "If all worlds contain a single deity, we can talk about the number one in non-theological contexts".
The obvious theist counter-reply is that the structure of God's desires is logically related to the essence of God, in a way that you can't have the goodness without the God nor more than God without the goodness, they are part of the same logical structure. (Aquinas: "God is by essence goodness itself")
I think this is a self-consistent metaethics as metaethics goes. The problem is that God is at the same time part of the realm of abstract logical structures like "goodness", and a concrete being that causes the world to exist, causes miracles, has desires, etc. The fault is not in the metaethics, it is in the confused metaphysics that allows for a concrete being to "exist essentially" as part of its logical structure.
ETA: of course, you could say the metaethics is self-consistent but also false, because it locates "goodness" outside ourselves (our extrapolated desires) which is where it really is. But for the Thomist I am currently emulating, "our extrapolated desires" sound a lot like "our final cause, the perfection to which we tend by our essence" and God is the ultimate final cause. The problem is again the metaphysics (in this case, using final causes without realizing they are mind projecting fallacy), not the metaethics.
My mind reduces all of this to "God = Confusion". What am I missing?
Well, I said that the metaphysics is confused, so we agree. I just think the metaethics part of religious philosophy can be put in order without falling into Euthyphro, the problem is in its broader philosophical system.
Not quite how I'd put it. I meant that in my mind the whole metaethics part implies that "God" is just a shorthand term for "whatever turns out to be 'goodness', even if we don't understand it yet", and that this resolves to the fact that "God" serves no other purposes than to confuse morality with other things within this context.
I think we still agree, though.
Using the word also implies that this goodness-embodying thing is sapient and has superpowers.
Or that it is sometimes useful to tell metaphorical stories about this goodness-embodying thing as if it were sapient and had superpowers.
Or as if the ancients thought it was sapient and had superpowers. They were wrong about that, but right about enough important things that we still value their writings.
As I explained here, it's perfectly reasonable to describe mathematical abstractions as causes.
I think that's pretty close to what a lot of religious people actually believe in. They just like the one-syllable description.
It seems like you're claiming an identity relationship between god and morality, and I find myself very confused as to what that could possibly mean.
I mean, it's sort of like I just encountered someone claiming that "friendship" and "dolphins" are really the same thing. One or both of us must be very confused about what the labels "friendship" and/or "dolphins" signify, or what this idea of "sameness" is, or something else...
See Alejandro's comment. Define G-d as "that which creates morality, and also lives in the sky and has superpowers". If you insist on the view of morality as a fixed logical abstraction, that would be a set of axioms. (Modus ponens has the Buddha-nature!) Then all you have to do is settle the factual question of whether the short-tempered creator who ordered you to genocide your neighbors embodies this set of axioms. If not, well, you live in a weird hybrid universe where G-d intervened to give you some sense of morality but is weaker than whichever Cthulhu or amoral physical law made and rules your world. Sorry.
Out of curiosity, why do you write G-d, not God? The original injunction against taking God's name in vain applied to the name in the old testament, which is usually mangled in the modern English as Jehovah, not to the mangled Germanic word meaning "idol".
People who care about that kind of thing usually think it counts as a Name, but don't think there's anything wrong with typing it (though it's still best avoided in case someone prints out the page). Trying to write it makes me squirm horribly and if I absolutely need the whole word I'll copy-paste it. I can totally write small-g "god" though, to talk about deities in general (or as a polite cuss). I feel absolutely silly about it, I'm an atheist and I'm not even Jewish (though I do have a weird cultural-appropriatey obsession). Oh well, everyone has weird phobias.
How interesting. Phobias are a form of alief, which makes this oddly relevant to my recent post.
I don't think it's quite the same. I have these sinking moments of "Whew, thank... wait, thank nothing" and "Oh please... crap, nobody's listening", but here I don't feel like I'm being disrespectful to Sky Dude (and if I cared I wouldn't call him Sky Dude). The emotion is clearly associated with the word, and doesn't go "whoops, looks like I have no referent" upon reflection.
What seems to be behind it is a feeling that if I did that, I would be practicing my religion wrong, and I like my religion. It's a jumble of things that give me an oxytocin kick, mostly consciously picked up, but it grows organically and sometimes plucks new dogma out of the environment. ("From now on Ruby Tuesday counts as religious music. Any questions?") I can't easily shed a part, it has to stop feeling sacred of its own accord.
You can eliminate inconvenient phobias with flooding. I can personally recommend sacrilege.
EDIT: It sounds like maybe it's not just a phobia.
I think there's a bug in your theist-simulation module ^^
I've yet to meet one that could have spontaneously come up with that statement.
Anyway, more to the point... in the definition of god you give, it seems to me that the "lives in sky with superpowers" part is sort of tacked on to the "creates morality" part, and I don't see why I can't talk about the "creates morality" part separate from the tacked-on bits. And if that is possible, I think this definition of god is still vulnerable to the dilemma (although it would seem clear that the second horn is the correct one; god contains a perfect implementation of morality, therefore what he says happens to be moral).
Hi there.
Are you a real theist or do you just like to abuse the common terminology (like, as far as I can tell, user:WillNewsome)? :)
This is a classic case of fighting the wrong battle against theism. The classic theist defence is to define away every meaningful aspect of God, piece by piece, until the question of God's existance is about as meaningful as asking "do you believe in the axiom of choice?". Then, after you've failed to disprove their now untestable (and therefore meaningless) theory, they consider themselves victorious and get back to reading the bible. It's this part that's the weak link. The idea that the bible tells us something about God (and therefore by extension morality and truth) is a testable and debatable hypothesis, whereas God's existance can be defined away into something that is not.
People can say "morality is God's will" all they like and I'll just tell them "butterflies are schmetterlinge". It's when they say "morality is in the bible" that you can start asking some pertinent questions. To mix my metaphors, I'll start believing when someone actually physically breaks a ball into pieces and reconstructs them into two balls of the same original size, but until I really see something like that actually happen it's all just navel gazing.
Sure, and to the extent that somebody answers that way, or for that matter runs away from the question, instead of doing that thing where they actually teach you in Jewish elementary school that Abraham being willing to slaughter Isaac for God was like the greatest thing ever and made him deserve to be patriarch of the Jewish people, I will be all like, "Oh, so under whatever name, and for whatever reason, you don't want to slaughter children - I'll drink to that and be friends with you, even if the two of us think we have different metaethics justifying it". I wasn't claiming that accepting the first horn of the dilemma was endorsed by all theists or a necessary implication of theism - but of course, the rejectance of that horn is very standard atheism.
I don't think it's incompatible. You're supposed to really trust the guy because he's literally made of morality, so if he tells you something that sounds immoral (and you're not, like, psychotic) of course you assume that it's moral and the error is on your side. Most of the time you don't get direct exceptional divine commands, so you don't want to kill any kids. Wouldn't you kill the kid if an AI you knew to be Friendly, smart, and well-informed told you "I can't tell you why right now, but it's really important that you kill that kid"?
If your objection is that Mr. Orders-multiple-genocides hasn't shown that kind of evidence he's morally good, well, I got nuthin'.
What we have is an inconsistent set of four assertions:
At least one of these has to be rejected. Abraham (provisionally) rejects 1; once God announces 'J/K,' he updates in favor of rejecting 2, on the grounds that God didn't really want him to kill his son, though the Voice really was God.
The problem with this is that rejecting 1 assumes that my confidence in my foundational moral principles (e.g., 'thou shalt not murder, self!') is weaker than my confidence in the conjunction of:
But it's hard to believe that I'm more confident in the divinity of a certain class of Voices than in my moral axioms, especially if my confidence in my axioms is what allowed me to conclude 4 (God/morality identity of some sort) in the first place. The problem is that I'm the one who has to decide what to do. I can't completely outsource my moral judgments to the Voice, because my native moral judgments are an indispensable part of my evidence for the properties of the Voice (specifically, its moral reliability). After all, the claim is 'God is perfectly moral, therefore I should obey him,' not 'God should be obeyed, therefore he is perfectly moral.'
The problem has the same structure for MixedNuts' analogy of the FAI replacing the Voice. Suppose you program the AI to compute explicitly the logical structure "morality" that EY is talking about, and it tells you to kill a child. You could think you made a mistake in the program (analogous to rejecting your 3), or that you are misunderstanding the AI or hallucinating it (rejecting 2). And in fact for most conjunctions of reasonable empirical assumptions, it would be more rational to take any of these options than to go ahead and kill the child.
Likewise, sensible religionists agree that if someone hears voices in their head telling them to kill children, they shouldn't do it. Some of they might say however that Abraham's position was unique, that he had especially good reasons (unspecified) to accept 2 and 3, and that for him killing the child is the right decision. In the same way, maybe an AI programmer with very strong evidence for the analogies for 2 and 3 should go ahead and kill the child. (What if the AI has computed that the child will grow up to be Hitler?)
A few religious thinkers (Kierkegaard) don't think Abraham's position was completely unique, and do think we should obey certain Voices without adequate evidence for 4, perhaps even without adequate evidence for 3. But these are outlier theories, and certainly don't reflect the intuitions of most religious believers, who pay more lip service to belief-in-belief than actual service-service to belief-in-belief.
I think an analogous AI set-up would be:
What you call rejecting 3 is closer to rejecting 4, since it concerns my confidence that the AI is moral, not my confidence that the AI I programmed is the same as the entity outputting 'Kill your son.'
I disagree, because I think the analogy between the (4) of each case should go this way:
(4a) Analysis of "morality" as equivalent to a logical structure extrapolatable from by brain state (plus other things) and that an AI can in principle compute <==> (4b) Analysis of "morality" as equivalent to a logical structure embodied in a unique perfect entity called "God"
These are both metaethical theories, a matter of philosophy. Then the analogy between (3) in each case goes:
(3a) This AI in front of me is accurately programmed to compute morality and display what I ought to do <==> (3b) This voice I hear is the voice of God telling me what I ought to do.
(3a) includes both your 3 and your 4, which can be put together as they are both empirical beliefs that, jointly, are related to the philosophical theory (4a) as the empirical belief (3b) is related to the philosophical theory (4b).
Makes sense. I was being deliberately vague about (4) because I wasn't committing myself to a particular view of why Abraham is confident in God's morality. If we're going with the scholastic, analytical, logical-pinpointing approach, then your framework is more useful. Though in that case even talking about 'God' or a particular AI may be misleading; what 4 then is really asserting is just that morality is a coherent concept, and can generate decision procedures. Your 3 is then the empirical claim that a particular being in the world embodies this concept of a perfect moral agent. My original thought simply took your 4 for granted (if there is no such concept, then what are we even talking about?), and broke the empirical claim up into multiple parts. This is important for the Abraham case, because my version of 3 is the premise most atheists reject, whereas there is no particular reason for the atheists to reject my version of 4 (or yours).
We are mostly in agreement about the general picture, but just to keep the conversation going...
I don't think (4) is so trivial or that (4a) and (4b) can be equated. For the first, there are other metaethical theories that I think wouldn't agree with the common content of (4a) and (4b). These include relativism, error theory, Moorean non-naturalism, and perhaps some naive naturalisms ("the good just is pleasure/happiness/etc, end of story").
For the second, I was thinking of (4a) as embedded in the global naturalistic, reductionistic philosophical picture that EY is elaborating and that is broadly accepted in LW, and of (4b) as embedded in the global Scholastic worldview (the most steelmanned version I know of religion). Obviously there are many differences between the two philosophies, both in the conceptual structures used and in very general factual beliefs (which as a Quinean I see as intertwined and inseparable at the most global level). In particular, I intended (4b) to include the claim that this perfect entity embodying morality actually exists as a concrete being (and, implicitly,that it has the other omni-properties attributed to God). Clearly atheists wouldn't agree with any of this.
Well, deities should make themselves clear enough that (2) is very likely (maybe the voice is pulling your leg, but it wants you to at least get started on the son-killing). (3) is also near-certain because you've had chats with this voice for decades, about moving and having kids and changing your name and whether the voice should destroy a city.
So this correctly tests whether you believe (4) more than (1) - whether your trust in G-d is greater than your confidence in your object-level judgement.
You're right that it's not clear why Abraham believes or should believe (4). His culture told him so and the guy has mostly done nice things for him and his wife, and promised nice things then delivered, but this hardly justifies blind faith. (Then again I've trusted people on flimsier grounds, if with lower stakes.) G-d seems very big on trust so it makes sense that he'd select the president of his fan club according to that criterion, and reinforce the trust with "look, you trusted me even though you expected it to suck, and it didn't suck".
Well, if we're shifting from our idealized post-Protestant-Reformation Abraham to the original Abraham-of-Genesis folk hero, then we should probably bracket all this Medieval talk about God's omnibenevolence and omnipotence. The Yahweh of Genesis is described as being unable to do certain things, as lacking certain items of knowledge, and as making mistakes. Shall not the judge of all the Earth do right?
As Genesis presents the story, the relevant question doesn't seem to be 'Does my moral obligation to obey God outweigh my moral obligation to protect my son?' Nor is it 'Does my confidence in my moral intuitions outweigh my confidence in God's moral intuitions plus my understanding of God's commands?' Rather, the question is: 'Do I care more about obeying God than about my most beloved possession?' Notice there's nothing moral at stake here at all; it's purely a question of weighing loyalties and desires, of weighing the amount I trust God's promises and respect God's authority against the amount of utility (love, happiness) I assign to my son.
The moral rights of the son, and the duties of the father, are not on the table; what's at issue is whether Abraham's such a good soldier-servant that he's willing to give up his most cherished possessions (which just happen to be sentient persons). Replace 'God' with 'Satan' and you get the same fealty calculation on Abraham's part, since God's authority, power, and honesty, not his beneficence, are what Abraham has faith in.
I can't speak for Jewish elementary school, but surely believing PA (even when, intuitively, the result seems flatly wrong or nonsensical) would be a good example to hold up before students of mathematics? The Monty Hall problem seems like a good example of this.
Is Schmidhuber's formalization of elegance the sort of thing you are seeking to do with rightness?
That's the default with no additional data, but I would hesitate, because to me how much each of the persons need the pie is also important in defining "fairness". If one of the three is starving while the others two are well-fed, it would be fair to give more to the one starving.
It may be just nitpicking, but since you took the point to ensure there is no difference between the three characters are involved in spotting the pie, but not mentioned they have the same need of it, it may pinpoint a deeper difference between different conceptions of "fairness" (should give them two different names ?)
Yay, I think we've finished the prerequisites to prerequisites, and started the prerequisites!
Scott Adams on the same subject, the morning after your post:
[...]
I don't like the idea of the words I use having definitions that I am unaware of and even after long reflection cannot figure out - not just the subtleties and edge cases, but massive central issues.
Don't like in the sense of considering this an annoying standard flaw in human minds or don't think this is correct? Where possible, the flaw is fixed by introducing more explicit definitions, and using those definitions instead of the immensely complicated and often less useful naive concepts. For human motivation, this doesn't seem to work particularly well.
That's precisely why I think motivation is not really a kind of definition. There are analogies, sure, but definitions are the things that you can change whenever you feel like it, but motivations are not.
I think that addressing metaethics conceptually, or as you would say logically, is the right way to go. I also think that doing so is just the kind of armchair conceptual analysis that philosphers go in for, and which is regularly criticised here.
The claim seems to be that moral judgement--first-order, not metaethical--is purely logical, but the justification ("grinding up the universe") only seems to go as far as showing it to be necessarily partly logical. And first-order ethics clearly has empirical elements. If human biology was such as to lay eggs and leave them to fend for themselves, there would be no immorailty in "child neglect".
I don't think there is clear route from "we can figure out morality ourselves" to "we can stop telling lies to children". The problem is that once you know morality is in a sense man-made, it becomes tempting to remake it self-servingly. I think we tell ourselves stories that fundamental morality comes from God Or Nature to restrain ourselves, and partly forget its man made nature. Men are not created equal, but it we believe they are, we behave better. "Created equal" is a value masquerading as a fact.
Great post! I agree with your analysis of moral semantics.
However, the question of moral ontology remains...do objective moral values exist? Is there anything I (or anyone) should do, independent from what I desire? With such a clear explanation of moral semantics at hand, I think the answer is an obvious and resounding no. Why would we even think that this is the case? One conclusion we can draw from this post is that telling an unfriendly AI that what it's doing is "wrong" won't affect its behavior. Because that which is "wrong" might be exactly that which is "moreclippy"! I feel that Eliezer probably agrees with me, here, since I gained I lot of insight into the issue from reading Three Worlds Collide.
Asking why we value that which is "right" is a scientific question, with a scientific answer. Our values are what they are, now, though, so, minus the semantics, doesn't morality just reduce to decision theory?
That being the case, what is it about us that makes us care about "rightness" then? What reason do you have for believing that the logical truth of what is right will has more influence on human behaviour than it would on any other general intelligence?
Certainly I can agree that there's reasons to worry another intelligence might not care about what's "right", since not every human really cares that much about it either. But it feels like your expected level of caring is "not at all", whereas my expected level of caring is "about as much as we do". Don't get me wrong, the variance in my estimate and the risk involved is still enough to justify the SI and its work. I just wonder about the difference between the two estimates.
I think your discussions of metaethics might be improved by rigorously avoiding words like "fair," "right," "better," "moral," "good," etc. I like the idea that "fair" points to a logical algorithm whose properties we can discuss objectively, but when you insist on using the word "fair," and no other word, as your pointer to this algorithm, people inevitably get confused. It seems like you are insisting that words have objective meanings, or that your morality is universally compelling, or something. You can and do explicitly deny these, but when you continue to rely exclusively on the word "fair" as if there is only one concept that that word can possibly point to, it's not clear what your alternative is.
Whereas if you use different symbols as pointers to your algorithms, the message (as I understand it) becomes much clearer. Translate something like:
Fair is dividing up food equally. Now, is dividing up the pie equally objectively fair? Yes: someone who wants to divide up the pie differently is talking about something other than fairness. So the assertion "dividing the pie equally is fair" is objectively true.
into
Define XYZZY as the algorithm "divide up food equally." Now, is dividing up the pie equally objectively XYZZY? Of course it is: that's a direct logical consequence of how I just defined XYZZY. Someone who wants to divide the pie differently is using an algorithm that is not XYZZY. The assertion "dividing up the pie equally is XYZZY" is as objective as the assertion "S0+S0=SS0"--someone who rejects the latter is not doing Peano arithmetic. By the way, when I personally say the word "fair," I mean "XYZZY."
I suspect that wording things like this has less potential to trip people up: it's much easier to reason logically about XYZZY than about fairness, even if both words are supposed to be pointers to the same concept.
I still feel confused. I definitely see that, when we talk about fairness, our intended meaning is logical in nature. So, if I claim that it is fair for each person to get an equal share of pie, I'm trying to talk about some set of axioms and facts derived from them. Trying.
The problem is, I'm not convinced that the underlying cognitive algorithms are stable enough for those axioms to be useful. Imagine, for example, a two-year-old with the usual attention span. What they consider "good" might vary quite quickly. What I consider "just" probably depends on how recently I ate. Even beyond such simple time dependence, what I consider "just" will definitely depend on context, framing, and how you measure my opinion (just ask a question? Show me a short film? Simulate the experience and see if I intervene?). Part of why friendly AI is so hard is that humans aren't just complicated, we're not even consistent. How, then, can we axiomatize a real human's idea of "justice" in a useful way?
I think elegance has to invoke agent-dependent concepts, because I think it's a composite description, which involves intuitive managing of agents, or descriptions of agents, rather. Intuitively it feels like something to be described as elegant requires "an intent" or a goal - and goal seems like it requires a description which involves some thingy trying to do something - that is met in a way that meets some comparative criterion, that this particular thing that met that particular goal, was somehow different from your expectations.
In otherwords it might not even be possible to define elegance without invoking agent-related concepts. (Or maybe it's just whatever conception of elegance I had)
I think that's a misapplication of reductionism (the thing I think Eliezer is thinking about he said it was mistakenly), where people take something they've logically attached to a value, and then reduce it to something else, which starts to feel like they can't reattach it to whatever they thought had the value in the first place.
To make an example it could be said that action A leads to result Y, and that result Y feels like a good thing, so action A feels like a good thing to do. So the person reduces their map of action A leading to result Y so that it no longer contains these things they associate into their feelings or values, because they momentarily look different. Then they can no longer associate action A with the feeling/value they had associatd result Y into, and it feels like action A can't be "moral" or "good" or whatever. (like, if you imagine "atoms bouncing around" instead of "giving food to starving people")
I think this tendency is also linked, sometimes at least, to people's mistake avoiding hesitancy. Or rather having a cautious way of doing things because of wanting to avoid mistakes. So in order to avoid making the mistake of being immoral, you wan't to be able to logically derive moral or immoral actions, and since morality seems reducible to nothing, it seems that this task is not possible. Kind of like you want to double check on your actions objectively, and when you hit this point of failure, it feels like you can't take the actions themselves, because you're used to doing actions this way. But anyway that was just random speculation and it's probably nonsense. Also I didn't mean to "box away" people's habits. I think it's often very useful to be cautious.
I think that reductionism, when misunderstood, can make the world look like a bucketful of nihilistic goo. Especially if it's used to devalue.
Clippy would encounter "ethical" dilemmas of the sort: Is it better ..err.. moreclippy to have 1 big paperclip, or 3 small paperclips? A line of many clips? Or a big clip made of smaller clips? Is it moreclippy to have 10 clips today and 20 clips tomorrow, or, 0 clips today and 30 clips tomorrow?
Just joking.. :)
edit: added " " to ethical
Having settled the meta-ethics, will you have anything to say about the ethics? Concrete theorems, with proofs, about how we should live?
To derive an ethic from a metaethic, I think you need to plug in a parameter that describes the entire context of human existence. Metaethic(Context) -> Ethic
So I don't know what you expect such a "theorem" and such "proofs" to look like, without containing several volumes descriptive in symbolic form of the human context.
I have no such expectation either. But I do expect something, for what use is meta-ethics if no ethics results, or at least, practical procedures for discovering ethics?
What do you have in mind by "a description in symbolic form of the human context"? The Cyc database? What would you do with it?
We have the processing unit called "brain" which does contain our understanding of the human context and therefore can plug a context parameter into a metaethical philosophy and thus derive an ethic. But we can't currently express the functioning of the brain as theorems and proofs -- our understanding of its working is far fuzzier than that.
I expect that the use of metaethic in AI development would similarly be so that the AI has something to plug its understanding of the human context into.
It hasn't been established that we can't have them, just that we can't by some formal, computational method.I'm afraid we're back to hand-wavy socio-politico-philosphical discussion.