= 27d567bc2d48d9f40e9ccc20ea370b15)
davidpearce comments on Decision Theory FAQ - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (467)
But note burger-choosing Jane (6.1) is still irrational - for she has discounted the much stronger preference of a cow not to be harmed. Rationality entails overcoming egocentric bias - and ethnocentric and anthropocentric bias - and adopting a God's eye point-of-view that impartially gives weight to all possible first-person perspectives.
When we say 'rationality', we mean instrumental rationality; getting what you want. Elsewhere, we also refer to epistemic rationality, which is believing true things. In neither case do we say anything about what you should want.
It might be a good thing to care about cows, but it's not rationality as we understand the word. Good you bring this up though, as I can easily imagine others being confused.
See also What Do We Mean by Rationality
This begs the question against moral realism. If it is in fact true that harming cows is bad, then epistemic rationality demands that we believe that harming cows is bad. Of course, saying that you should believe harming cows is morally wrong is different from saying that you shouldn't choose to harm cows, but the inference from one to the other is pretty plausible. It seems fairly uncontroversial that if one believes that action X is bad, and it is in fact true that action X is bad, then one should not perform action X (ceteris paribus).
I don't agree with davidpearce's framing (that rationality demands that one give equal weight to all perspectives), but I also don't agree with the claim that rationality does not tell us anything about what we should want. Perhaps instrumental rationality doesn't, but epistemic rationality does.
Sure - but rationality doesn't itself imply moral conclusions. It only says that if there are moral facts then we should believe them, not that there are any (particular) moral facts, which is what David needs.
Rationality may imply moral conclusions in the same sense that it implies some factual conclusions: we think that folks who believe in creationism are irrational, because we think the evidence for evolution is sufficiently strong and also think that evolution is incompatible with creationism. Analogously, if the evidence for some moral truth is sufficiently strong, we may similarly accuse of irrationality those who fail to form their beliefs accordingly. So it is misleading to say that "rationality doesn't itself imply moral conclusions".
Related discussion: http://lesswrong.com/lw/gnb/questions_for_moral_realists/8ls8?context=3
Larks, no, pragmatist nicely captured the point I was making. If we are trying to set out what an "ideal, perfectly rational agent" would choose, then we can't assume that such a perfectly rational agent would arbitrarily disregard a stronger preference in favour of a weaker preference. Today, asymmetries of epistemic access mean that weaker preferences often trump stronger preferences; but with tomorrow's technology, this cognitive limitation on our decision-making procedures can be overcome.
Isn't the giant elephant in this room the whole issue of moral realism? I'm a moral cognitivist but not a moral realist. I have laid out what it means for my moral beliefs to be true - the combination of physical fact and logical function against which my moral judgments are being compared. This gives my moral beliefs truth value. And having laid this out, it becomes perfectly obvious that it's possible to build powerful optimizers who are not motivated by what I call moral truths; they are maximizing something other than morality, like paperclips. They will also meta-maximize something other than morality if you ask them to choose between possible utility functions, and will quite predictably go on picking the utility function "maximize paperclips". Just as I correctly know it is better to be moral than to be paperclippy, they accurately evaluate that it is more paperclippy to maximize paperclips than morality. They know damn well that they're making you unhappy and violating your strong preferences by doing so. It's just that all this talk about the preferences that feel so intrinsically motivating to you, is itself of no interest to them because you haven't gotten to the all-important parts about paperclips yet.
The main thing I'm not clear on in this discussion is to what extent David Pearce is being innocently mysterian vs. motivatedly mysterian. To be confused about how your happiness seems so intrinsically motivating, and innocently if naively wonder if perhaps it must be intrinsically motivating to other minds as well, is one thing. It is another thing to prefer this conclusion and so to feel a bit uncurious about anyone's detailed explanation of how it doesn't work like that. It is even less innocent to refuse outright to listen when somebody else tries to explain. And then strangest of all is to state powerfully and definitely that every bit of happiness must be motivating to all other minds, even though you can't lay out step by step how the decision procedure would work. This requires overrunning your own claims to knowledge in a fundamental sense - mistaking your confusion about something for the ability to make definite claims about it. Now this of course is a very common and understandable sin, and the fact that David Pearce is crusading for happiness for all life forms should certainly count into our evaluation of his net virtue (it would certainly make me willing to drink a Pepsi with him). But I'm also not clear about where to go from here, or whether this conversation is accomplishing anything useful.
In particular it seems like David Pearce is not leveling any sort of argument we could possibly find persuasive - it's not written so as to convince anyone who isn't already a moral realist, or addressing the basic roots of disagreement - and that's not a good sign. And short of rewriting the entire metaethics sequence in these comments I don't know how I could convince him, either.
Eliezer, in my view, we don't need to assume meta-ethical realism to recognise that it's irrational - both epistemically irrational and instrumentally irrational - arbitrarily to privilege a weak preference over a strong preference. To be sure, millions of years of selection pressure means that the weak preference is often more readily accessible. In the here-and-now, weak-minded Jane wants a burger asap. But it's irrational to confuse an epistemological limitation with a deep metaphysical truth. A precondition of rational action is understanding the world. If Jane is scientifically literate, then she'll internalise Nagel's "view from nowhere" and adopt the God's-eye-view to which natural science aspires. She'll recognise that all first-person facts are ontologically on a par - and accordingly act to satisfy the stronger preference over the weaker. So the ideal rational agent in our canonical normative decision theory will impartially choose the action with the highest expected utility - not the action with an extremely low expected utility. At the risk of labouring the obvious, the difference in hedonic tone induced by eating a hamburger and a veggieburger is minimal. By contrast, the ghastly experience of having one's throat slit is exceptionally unpleasant. Building anthropocentric bias into normative decision theory is no more rational than building geocentric bias into physics.
Paperclippers? Perhaps let us consider the mechanism by which paperclips can take on supreme value. We understand, in principle at least, how to make paperclips seem intrinsically supremely valuable to biological minds - more valuable than the prospect of happiness in the abstract. [“Happiness is a very pretty thing to feel, but very dry to talk about.” - Jeremy Bentham]. Experimentally, perhaps we might use imprinting (recall Lorenz and his goslings), microelectrodes implanted in the reward and punishment centres, behavioural conditioning and ideological indoctrination - and perhaps the promise of 72 virgins in the afterlife for the faithful paperclipper. The result: a fanatical paperclip fetishist! Moreover, we have created a full-spectrum paperclip -fetishist. Our human paperclipper is endowed, not merely with some formal abstract utility function involving maximising the cosmic abundance of paperclips, but also first-person "raw feels" of pure paperclippiness. Sublime!
However, can we envisage a full-spectrum paperclipper superintelligence? This is more problematic. In organic robots at least, the neurological underpinnings of paperclip evangelism lie in neural projections from our paperclipper's limbic pathways - crudely, from his pleasure and pain centres. If he's intelligent, and certainly if he wants to convert the world into paperclips, our human paperclipper will need to unravel the molecular basis of the so-called "encephalisation of emotion". The encephalisation of emotion helped drive the evolution of vertebrate intelligence - and also the paperclipper's experimentally-induced paperclip fetish / appreciation of the overriding value of paperclips. Thus if we now functionally sever these limbic projections to his neocortex, or if we co-administer him a dopamine antagonist and a mu-opioid antagonist, then the paperclip-fetishist's neocortical representations of paperclips will cease to seem intrinsically valuable or motivating. The scales fall from our poor paperclipper's eyes! Paperclippiness, he realises, is in the eye of the beholder. By themselves, neocortical paperclip representations are motivationally inert. Paperclip representations can seem intrinsically valuable within a paperclipper's world-simulation only in virtue of their rewarding opioidergic projections from his limbic system - the engine of phenomenal value. The seemingly mind-independent value of paperclips, part of the very fabric of the paperclipper's reality, has been been unmasked as derivative. Critically, an intelligent and recursively self-improving paperclipper will come to realise the parasitic nature of the relationship between his paperclip experience and hedonic innervation: he's not a naive direct realist about perception. In short, he'll mature and acquire an understanding of basic neuroscience.
Now contrast this case of a curable paperclip-fetish with the experience of e.g. raw phenomenal agony or pure bliss - experiences not linked to any fetishised intentional object. Agony and bliss are not dependent for their subjective (dis)value on anything external to themselves. It's not an open question (cf. http://en.wikipedia.org/wiki/Open-question_argument) whether one's unbearable agony is subjectively disvaluable. For reasons we simply don't understand, first-person states on the pleasure-pain axis have a normative aspect built into their very nature. If one is in agony or despair, the subjectively disvaluable nature of this agony or despair is built into the nature of the experience itself. To be panic-stricken, to take another example, is universally and inherently disvaluable to the subject whether one is a fish or a cow or a human being.
Why does such experience exist? Well, I could speculate and tell a naturalistic reductive story involving Strawsonian physicalism (cf. http://en.wikipedia.org/wiki/Physicalism#Strawsonian_physicalism) and possible solutions to the phenomenal binding problem (cf. http://cdn.preterhuman.net/texts/body_and_health/Neurology/Binding.pdf). But to do so here opens a fresh can of worms.
Eliezer, I understand you believe I'm guilty of confusing an idiosyncratic feature of my own mind with a universal architectural feature of all minds. Maybe so! As you say, this is a common error. But unless I'm ontologically special (which I very much doubt!) the pain-pleasure axis discloses the world's inbuilt metric of (dis)value - and it's a prerequisite of finding anything (dis)valuable at all.
You need some stage at which a fact grabs control of a mind, regardless of any other properties of its construction, and causes its motor output to have a certain value.
As Sarokrae observes, this isn't the idea at all. We construct a paperclip maximizer by building an agent which has a good model of which actions lead to which world-states (obtained by a simplicity prior and Bayesian updating on sense data) and which always chooses consequentialistically the action which it expects to lead to the largest number of paperclips. It also makes self-modification choices by always choosing the action which leads to the greatest number of expected paperclips. That's all. It doesn't have any pleasure or pain, because it is a consequentialist agent rather than a policy-reinforcement agent. Generating compressed, efficient predictive models of organisms that do experience pleasure or pain, does not obligate it to modify its own architecture to experience pleasure or pain. It also doesn't care about some abstract quantity called "utility" which ought to obey logical meta-properties like "non-arbitrariness", so it doesn't need to believe that paperclips occupy a maximum of these meta-properties. It is not an expected utility maximizer. It is an expected paperclip maximizer. It just outputs the action which leads to the maximum number of expected paperclips. If it has a very powerful and accurate model of which actions lead to how many paperclips, it is a very powerful intelligence.
You cannot prohibit the expected paperclip maximizer from existing unless you can prohibit superintelligences from accurately calculating which actions lead to how many paperclips, and efficiently searching out plans that would in fact lead to great numbers of paperclips. If you can calculate that, you can hook up that calculation to a motor output and there you go.
Yes, this is a prospect of Lovecraftian horror. It is a major problem, kind of the big problem, that simple AI designs yield Lovecraftian horrors.
Eliezer, thanks for clarifying. This is how I originally conceived you viewed the threat from superintelligent paperclip-maximisers, i.e. nonconscious super-optimisers. But I was thrown by your suggestion above that such a paperclipper could actually understand first-person phenomenal states, i.e, it's a hypothetical "full-spectrum" paperclipper. If a hitherto non-conscious super-optimiser somehow stumbles upon consciousness, then it has made a momentous ontological discovery about the natural world. The conceptual distinction between the conscious and nonconscious is perhaps the most fundamental I know. And if - whether by interacting with sentients or by other means - the paperclipper discovers the first-person phenomenology of the pleasure-pain axis, then how can this earth-shattering revelation leave its utility function / world-model unchanged? Anyone who is isn't profoundly disturbed by torture, for instance, or by agony so bad one would end the world to stop the horror, simply hasn't understood it. More agreeably, if such an insentient paperclip-maximiser stumbles on states of phenomenal bliss, might not clippy trade all the paperclips in the world to create more bliss, i.e revise its utility function? One of the traits of superior intelligence, after all, is a readiness to examine one's fundamental assmptions and presuppositions - and (if need be) create a novel conceptual scheme in the face of surprising or anomalous empirical evidence.
Similarly, anyone who doesn't want to maximize paperclips simply hasn't understood the ineffable appeal of paperclipping.
"Aargh!" he said out loud in real life. David, are you disagreeing with me here or do you honestly not understand what I'm getting at?
The whole idea is that an agent can fully understand, model, predict, manipulate, and derive all relevant facts that could affect which actions lead to how many paperclips, regarding happiness, without having a pleasure-pain architecture. I don't have a paperclipping architecture but this doesn't stop me from modeling and understanding paperclipping architectures.
The paperclipper can model and predict an agent (you) that (a) operates on a pleasure-pain architecture and (b) has a self-model consisting of introspectively opaque elements which actually contain internally coded instructions for your brain to experience or want certain things (e.g. happiness). The paperclipper can fully understand how your workspace is modeling happiness and know exactly how much you would want happiness and why you write papers about the apparent ineffability of happiness, without being happy itself or at all sympathetic toward you. It will experience no future surprise on comprehending these things, because it already knows them. It doesn't have any object-level brain circuits that can carry out the introspectively opaque instructions-to-David's-brain that your own qualia encode, so it has never "experienced" what you "experience". You could somewhat arbitrarily define this as a lack of knowledge, in defiance of the usual correspondence theory of truth, and despite the usual idea that knowledge is being able to narrow down possible states of the universe. In which case, symmetrically under this odd definition, you will never be said to "know" what it feels like to be a sentient paperclip maximizer or you would yourself be compelled to make paperclips above all else, for that is the internal instruction of that quale.
But if you take knowledge in the powerful-intelligence-relevant sense where to accurately represent the universe is to narrow down its possible states under some correspondence theory of truth, and to well model is to be able to efficiently predict, then I am not barred from understanding how the paperclip maximizer works by virtue of not having any internal instructions which tell me to only make paperclips, and it's not barred by its lack of pleasure-pain architecture from fully representing and efficiently reasoning about the exact cognitive architecture which makes you want to be happy and write sentences about the ineffable compellingness of happiness. There is nothing left for it to understand. This is also the only sort of "knowledge" or "understanding" that would inevitably be implied by Bayesian updating. So inventing a more exotic definition of "knowledge" which requires having completely modified your entire cognitive architecture just so that you can natively and non-sandboxed-ly obey the introspectively-opaque brain-instructions aka qualia of another agent with completely different goals, is not the sort of predictive knowledge you get just by running a powerful self-improving agent trying to better manipulate the world. You can't say, "But it will surely discover..."
I know that when you imagine this it feels like the paperclipper doesn't truly know happiness, but that's because, as an act of imagination, you're imagining the paperclipper without that introspectively-opaque brain-instructing model-element that you model as happiness, the modeled memory of which is your model of what "knowing happiness" feels like. And because the actual content and interpretation of these brain-instructions are introspectively opaque to you, you can't imagine anything except the quale itself that you imagine to constitute understanding of the quale, just as you can't imagine any configuration of mere atoms that seem to add up to a quale within your mental workspace. That's why people write papers about the hard problem of consciousness in the first place.
Even if you don't believe my exact account of the details, someone ought to be able to imagine that something like this, as soon as you actually knew how things were made of parts and could fully diagram out exactly what was going on in your own mind when you talked about happiness, would be true - that you would be able to efficiently manipulate models of it and predict anything predictable, without having the same cognitive architecture yourself, because you could break it into pieces and model the pieces. And if you can't fully credit that, you at least shouldn't be confident that it doesn't work that way, when you know you don't know why happiness feels so ineffably compelling!
Maybe I can chime in...
"understand" does not mean "empathize". Psychopaths understand very well when people experience these states but they do not empathize with them.
Again, understanding is insufficient for revision. The paperclip maximizer, like a psychopath, maybe better at parsing human affect than a regular human, but it is not capable of empathy, so it will manipulate this affect for its own purposes, be it luring a victim or building paperclips.
So, if one day humans discover the ultimate bliss that only creating paperclips can give, should they "create a novel conceptual scheme" of giving their all to building more paperclips, including converting themselves into metal wires? Or do we not qualify as a "superior intelligence"?
Was that claimed? The standard claim is that superintelligences can "model" other entities. That may not be enough to to understand qualia.
Pearce can prohibit paperclippers from existing by prohibiting superintelligences with narrow interests from existing. He doesn't have to argue that the clipper would not be able to instrumentally reason out how to make paperclips; Pearce can argue that to be a really good instrumental reasoner, an entity needs to have a very broad understanding, and that an entity with a broad understanding would not retain narrow interests.
(Edits for spelling and clarity)
To slightly expand, if an intelligence is not prohibited from the following epistemic feats:
1) Be good at predicting which hypothetical actions would lead to how many paperclips, as a question of pure fact.
2) Be good at searching out possible plans which would lead to unusually high numbers of paperclips - answering the purely epistemic search question, "What sort of plan would lead to many paperclips existing, if someone followed it?"
3) Be good at predicting and searching out which possible minds would, if constructed, be good at (1), (2), and (3) as purely epistemic feats.
Then we can hook up this epistemic capability to a motor output and away it goes. You cannot defeat the Orthogonality Thesis without prohibiting superintelligences from accomplishing 1-3 as purely epistemic feats. They must be unable to know the answers to these questions of fact.
Have to point out here that the above is emphatically not what Eliezer talks about when he says "maximise paperclips". Your examples above contain in themselves the actual, more intrisics values to which paperclips would be merely instrumental: feelings in your reward and punishment centres, virgins in the afterlife, and so on. You can re-wire the electrodes, or change the promise of what happens in the afterlife, and watch as the paperclip preference fades away.
What Eliezer is talking about is a being for whom "pleasure" and "pain" are not concepts. Paperclips ARE the reward. Lack of paperclips IS the punishment. Even if pleasure and pain are concepts, they are merely instrumental to obtaining more paperclips. Pleasure would be good because it results in paperclips, not vice versa. If you reverse the electrodes so that they stimulate the pain centre when they find paperclips, and the pleasure centre when there are no paperclips, this being would start instrumentally value pain more than pleasure, because that's what results in more paperclips.
It's a concept that's much more alien to our own minds than what you are imagining, and anthropomorphising it is rather more difficult!
Indeed, you touch upon this yourself:
Can you explain why pleasure is a more natural value than paperclips?
Minor correction: The mere post-factual correlation of pain to paperclips does not imply that more paperclips can be produced by causing more pain. You're talking about the scenario where each 1,000,000 screams produces 1 paperclip, in which case obviously pain has some value.
Sarokrae, first, as I've understood Eliezer, he's talking about a full-spectrum superintelligence, i.e. a superintelligence which understands not merely the physical processes of nociception etc, but the nature of first-person states of organic sentients. So the superintelligence is endowed with a pleasure-pain axis, at least in one of its modules. But are we imagining that the superintelligence has some sort of orthogonal axis of reward - the paperclippiness axis? What is the relationship between these dual axes? Can one grasp what it's like to be in unbearable agony and instead find it more "rewarding" to add another paperclip? Whether one is a superintelligence or a mouse, one can't directly access mind-independent paperclips, merely one's representations of paperclips. But what does it mean to say one's representation of a paperclip could be intrinsically "rewarding" in the absence of hedonic tone? [I promise I'm not trying to score some empty definitional victory, whatever that might mean; I'm just really struggling here...]
What Eliezer is talking about (a superintelligence paperclip maximiser) does not have a pleasure-pain axis. It would be capable of comprehending and fully emulating a creature with such an axis if doing so had a high expected value in paperclips but it does not have such a module as part of itself.
One of them it has (the one about paperclips). One of them it could, in principle, imagine (the thing with 'pain' and 'pleasure').
Yes. (I'm not trying to be trite here. That's the actual answer. Yes. Paperclip maximisers really maximise paperclips and really don't care about anything else. This isn't because they lack comprehension.)
Roughly speaking it means "It's going to do things that maximise paperclips and in some way evaluates possible universes with more paperclips as superior to possible universes with less paperclips. Translating this into human words we call this 'rewarding' even though that is inaccurate anthropomorphising."
(If I understand you correctly your position would be that the agent described above is nonsensical.)
Even among philosophers, "moral realism" is a term wont to confuse. I'd be wary about relying on it to chunk your philosophy. For instance, the simplest and least problematic definition of 'moral realism' is probably the doctrine...
minimal moral realism: cognitivism (moral assertions like 'murder is bad' have truth-conditions, express real beliefs, predicate properties of objects, etc.) + success theory (some moral assertions are true; i.e., rejection of error theory).
This seems to be the definition endorsed on SEP's Moral Realism article. But it can't be what you have in mind, since you accept cognitivism and reject error theory. So perhaps you mean to reject a slightly stronger claim (to coin a term):
factual moral realism: MMR + moral assertions are not true or false purely by stipulation (or 'by definition'); rather, their truth-conditions at least partly involve empirical, worldly contingencies.
But here, again, it's hard to find room to reject moral realism. Perhaps some moral statements, like 'suffering is bad,' are true only by stipulation; but if 'punching people in the face causes suffering' is not also true by stipulation, then the conclusion 'punching people in the face is bad' will not be purely stipulative. Similarly, 'The Earth's equatorial circumference is ~40,075.017 km' is not true just by definition, even though we need somewhat arbitrary definitions and measurement standards to assert it. And rejecting the next doesn't sound right either:
correspondence moral realism: FMR + moral assertions are not true or false purely because of subjects' beliefs about the moral truth. For example, the truth-condition for 'eating babies is bad' are not 'Eliezer Yudkowsky thinks eating babies is bad', nor even 'everyone thinks eating babies is bad'. Our opinions do play a role in what's right and wrong, but they don't do all the work.
So perhaps one of the following is closer to what you mean to deny:
moral transexperientialism: Moral facts are nontrivially sensitive to differences wholly independent of, and having no possible impact on, conscious experience. The goodness and badness of outcomes is not purely a matter of (i.e., is not fully fixed by) their consequences for sentients. This seems kin to Mark Johnston's criterion of 'response-dependence'. Something in this vicinity seems to be an important aspect of at least straw moral realism, but it's not playing a role here.
moral unconditionalism: There is a nontrivial sense in which a single specific foundation for (e.g., axiomatization of) the moral truths is the right one -- 'objectively', and not just according to itself or any persons or arbitrarily selected authority -- and all or most of the alternatives aren't the right one. (We might compare this to the view that there is only one right set of mathematical truths, and this rightness is not trivial or circular. Opposing views include mathematical conventionalism and 'if-thenism'.)
moral non-naturalism: Moral (or, more broadly, normative) facts are objective and worldly in an even stronger sense, and are special, sui generis, metaphysically distinct from the prosaic world described by physics.
Perhaps we should further divide this view into 'moral platonism', which reduces morality to logic/math but then treats logic/math as a transcendent, eternal Realm of Thingies and Stuff; v. 'moral supernaturalism', which identifies morality more with souls and ghosts and magic and gods than with logical thingies. If this distinction isn't clear yet, perhaps we could stipulate that platonic thingies are acausal, whereas spooky supernatural moral thingies can play a role in the causal order. I think this moral supernaturalism, in the end, is what you chiefly have in mind when you criticize 'moral realism', since the idea that there are magical, irreducible Moral-in-Themselves Entities that can exert causal influences on us in their own right seems to be a prerequisite for the doctrine that any possible agent would be compelled (presumably by these special, magically moral objects or properties) to instantiate certain moral intuitions. Christianity and karma are good examples of moral supernaturalisms, since they treat certain moral or quasi-moral rules and properties as though they were irreducible physical laws or invisible sorcerors.
At the same time, it's not clear that davidpearce was endorsing anything in the vicinity of moral supernaturalism. (Though I suppose a vestigial form of this assumption might still then be playing a role in the background. It's a good thing it's nearly epistemic spring cleaning time.) His view seems somewhere in the vicinity of unconditionalism -- if he thinks anyone who disregards the interests of cows is being unconditionally epistemically irrational, and not just 'epistemically irrational given that all humans naturally care about suffering in an agent-neutral way'. The onus is then on him and pragmatist to explain on what non-normative basis we could ever be justified in accepting a normative standard.
I'm not sure this taxonomy is helpful from David Pearce's perspective. David Pearce's position is that there are universally motivating facts - facts whose truth, once known, is compelling for every possible sort of mind. This reifies his observation that the desire for happiness feels really, actually compelling to him and this compellingness seems innate to qualia, so anyone who truly knew the facts about the quale would also know that compelling sense and act accordingly. This may not correspond exactly to what SEP says under moral realism and let me know if there's a standard term, but realism seems to describe the Pearcean (or Eliezer circa 1996) feeling about the subject - that happiness is really intrinsically preferable, that this is truth and not opinion.
From my perspective this is a confusion which I claim to fully and exactly understand, which licenses my definite rejection of the hypothesis. (The dawning of this understanding did in fact cause my definite rejection of the hypothesis in 2003.) The inherent-desirableness of happiness is your mind reifying the internal data describing its motivation to do something, so if you try to use your empathy to imagine another mind fully understanding this mysterious opaque data (quale) whose content is actually your internal code for "compelled to do that", you imagine the mind being compelled to do that. You'll be agnostic about whether or not this seems supernatural because you don't actually know where the mysterious compellingness comes from. From my perspective, this is "supernatural" because your story inherently revolves around mental facts you're not allowed to reduce to nonmental facts - any reduction to nonmental facts will let us construct a mind that doesn't care once the qualia aren't mysteriously irreducibly compelling anymore. But this is a judgment I pass from reductionist knowledge - from a Pearcean perspective, there's just a mysteriously compelling quality about happiness, and to know this quale seems identical with being compelled by it; that's all your story. Well, that plus the fact that anyone who says that some minds might not be compelled by happiness, seems to be asserting that happiness is objectively unimportant or that its rightness is a matter of mere opinion, which is obviously intuitively false. (As a moral cognitivist, of course, I agree that happiness is objectively important, I just know that "important" is a judgment about a certain logical truth that other minds do not find compelling. Since in fact nothing can be intrinsically compelling to all minds, I have decided not to be an error theorist as I would have to be if I took this impossible quality of intrinsic compellingness to be an unavoidable requirement of things being good, right, valuable, or important in the intuitive emotional sense. My old intuitive confusion about qualia doesn't seem worth respecting so much that I must now be indifferent between a universe of happiness vs. a universe of paperclips. The former is still better, it's just that now I know what "better" means.)
But if the very definitions of the debate are not automatically to judge in my favor, then we should have a term for what Pearce believes that reflects what Pearce thinks to be the case. "Moral realism" seems like a good term for "the existence of facts the knowledge of which is intrinsically and universally compelling, such as happiness and subjective desire". It may not describe what a moral cognitivist thinks is really going on, but "realism" seems to describe the feeling as it would occur to Pearce or Eliezer-1996. If not this term, then what? "Moral non-naturalism" is what a moral cognitivist says to deconstruct your theory - the self-evident intrinsic compellingness of happiness quales doesn't feel like asserting "non-naturalism" to David Pearce, although you could have a non-natural theory about how this mysterious observation was generated.
I'm not sure he's wrong in saying that feeling the qualia of a sentient, as opposed to modeling those qualia in an affective black box without letting the feels 'leak' into the rest of your cognitionspace, requires some motivational effect. There are two basic questions here:
First, the Affect-Effect Question: To what extent are the character of subjective experiences like joy and suffering intrinsic or internal to the state, as opposed to constitutively bound up in functional relations that include behavioral impetuses? (For example, to what extent is it possible to undergo the phenomenology of anguish without thereby wanting the anguish to stop? And to what extent is it possible to want something to stop without being behaviorally moved, to the extent one is able and to the extent one's other desires are inadequate overriders, to stop it?) Compare David Lewis' 'Mad Pain', pain that has the same experiential character as ordinary pain but none of its functional relations (or at least not the large-scale ones). Some people think a state of that sort wouldn't qualify as 'pain' at all, and this sort of relationalism lends some credibility to pearce's view.
Second, the Third-Person Qualia Question: To what extent is phenomenological modeling (modeling a state in such a way that you, or a proper part of you, experiences that state) required for complete factual knowledge of real-world agents? One could grant that qualia are real (and really play an important role in various worldly facts, albeit perhaps physical ones) and are moreover unavoidably motivating (if you aren't motivated to avoid something, then you don't really fear it), but deny that an epistemically rational agent is required to phenomenologically model qualia. Perhaps there is some way to represent the same mental states without thereby experiencing them, to fully capture the worldly facts about cows without simulating their experiences oneself. If so, then knowing everything about cows would not require one to be motivated (even in some tiny powerless portion of oneself) to fulfill the values of cows. (Incidentally, it's also possible in principle to grant the (admittedly spooky) claim that mental states are irreducible and indispensable, without thinking that you need to be in pain in order to fully and accurately model another agent's pain; perhaps it's possible to accurately model one phenomenology using a different phenomenology.)
And again, at this point I don't think any of these positions need to endorse supernaturalism, i.e., the idea that special moral facts are intervening in the causal order to force cow-simulators, against their will, to try to help cows. (Perhaps there's something spooky and supernatural about causally efficacious qualia, but for the moment I'll continue assuming they're physical states -- mayhap physical states construed in a specific way.) All that's being disputed, I think, is to what extent a programmer of a mind-modeler could isolate the phenomenology of states from their motivational or behavioral roles, and to what extent this programmer could model brains at all without modeling their first-person character.
As a limiting case: Assuming there are facts about conscious beings, could an agent simulate everything about those beings without ever becoming conscious itself? (And if it did become conscious, would it only be conscious inasmuch as it had tiny copies of conscious beings inside itself? Or would it also need to become conscious in a more global way, in order to access and manipulate useful information about its conscious subsystems?)
Incidentally, these engineering questions are in principle distinct both from the topic of causally efficacious irreducible Morality Stuff (what I called moral supernaturalism), and from the topic of whether moral claims are objectively right, that, causally efficacious or not, moral facts have a sort of 'glow of One True Oughtness' (what I called moral unconditionalism, though some might call it 'moral absolutism'), two claims the conjunction of which it sounds like you've been labeling 'moral realism', in deference to your erstwhile meta-ethic. Whether we can motivation-externally simulate experiential states with perfect fidelity and epistemic availability-to-the-simulating-system-at-large is a question for philosophy of mind and computer science, not for meta-ethics. (And perhaps davidpearce's actual view is closer to what you call moral realism than to my steelman. Regardless, I'm more interested in interrogating the steelman.)
So terms like 'non-naturalism' or 'supernaturalism' are too theory-laden and sophisticated for what you're imputing to Pearce (and ex-EY), which is really more of a hunch or thought-terminating-clichéplex. In that case, perhaps 'naïve (moral) realism' or 'naïve absolutism' is the clearest term you could use. (Actually, I like 'magical absolutism'. It has a nice ring to it, and 'magical' gets at the proto-supernaturalism while 'absolutism' gets at the proto-unconditionalism. Mm, words.) Philosophers love calling views naïve, and the term doesn't have a prior meaning like 'moral realism', so you wouldn't have to deal with people griping about your choice of jargon.
This would also probably be a smart rhetorical move, since a lot of people don't see a clear distinction between cognitivism and realism and might be turned off by your ideas qua an anti-realism theory even if they'd have loved them qua a realist theory. 'Tis part of why I tried to taboo the term as 'minimal moral realism' etc., rather than endorsing just one of the definitions on offer.
Eliezer, you remark, "The inherent-desirableness of happiness is your mind reifying the internal data describing its motivation to do something," Would you propose that a mind lacking in motivation couldn't feel blissfully happy? Mainlining heroin (I am told) induces pure bliss without desire - shades of Buddhist nirvana? Pure bliss without motivation can be induced by knocking out the dopamine system and directly administering mu opioid agonists to our twin "hedonic hotspots" in the ventral pallidum and rostral shell of the nucleus accumbens. Conversely, amplifying mesolimbic dopamine function while disabling the mu opioid pathways can induce desire without pleasure.
[I'm still mulling over some of your other points.]
Here we're reaching the borders of my ability to be confident about my replies, but the two answers which occur to me are:
1) It's not positive reinforcement unless feeling it makes you experience at least some preference to do it again - otherwise in what sense are the neural networks getting their plus? Heroin may not induce desire while you're on it, but the thought of the bliss induces desire to take heroin again, once you're off the heroin.
2) The superBuddhist no longer capable of experiencing desire or choice, even desire or choice over which thoughts to think, also becomes incapable of experiencing happiness (perhaps its neural networks aren't even being reinforced to make certain thoughts more likely to be repeated). However, you, who are still capable of desire and who still have positively reinforcing thoughts, might be tricked into considering the superBuddhist's experience to be analogous to your own happiness and therefore acquire a desire to be a superBuddhist as a result of imagining one - mostly on account of having been told that it was representing a similar quale on account of representing a similar internal code for an experience, without realizing that the rest of the superBuddhist's mind now lacks the context your own mind brings to interpreting that internal coding into pleasurable positive reinforcement that would make you desire to repeat that experiential state.
It's a reasonably good description, though wanting and liking seem to be neurologically separate, such that liking does not necessarily reflect a motivation, nor vice-versa (see: Not for the sake of pleasure alone. Think the pleasurable but non-motivating effect of opioids such as heroin. Even in cases in which wanting and liking occur together, this does not necessarily invalidate the liking aspect as purely wanting.
Liking and disliking, good and bad feelings as qualia, especially in very intense amounts, seem to be intrinsically so to those who are immediately feeling them. Reasoning could extend and generalize this.
Heh. Yes, I remember reading the section on noradrenergic vs. dopaminergic motivation in Pearce's BLTC as a 16-year-old. I used to be a Pearcean, ya know, hence the Superhappies. But that distinction didn't seem very relevant to the metaethical debate at hand.
Watch out -- the word "sentient" has at least two different common meanings, one of which includes cattle and the other doesn't. EY usually uses it with the narrower meaning (for which a less ambiguous synonym is "sapient"), whereas David Pearce seems to be using it with the broader meaning.
Ah. By 'sentient' I mean something that feels, by 'sapient' something that thinks.
To be more fine-grained about it, I'd define functional sentience as having affective (and perhaps perceptual) cognitive states (in a sense broad enough that it's obvious cows have them, and equally obvious tulips don't), and phenomenal sentience as having a first-person 'point of view' (though I'm an eliminativist about phenomenal consciousness, so my overtures to it above can be treated as a sort of extended thought experiment).
Similarly, we might distinguish a low-level kind of sapience (the ability to form and manipulate mental representations of situations, generate expectations and generalizations, and update based on new information) from a higher-level kind closer to human sapience (perhaps involving abstract and/or hyper-productive representations à la language).
Based on those definitions, I'd say it's obvious cows are functionally sentient and have low-level sapience, extremely unlikely they have high-level sapience, and unclear whether they have phenomenal sentience.
Rob, many thanks for a thoughtful discussion above. But on one point, I'm confused. You say of cows that it's "unclear whether they have phenomenal sentience." Are you using the term "sentience" in the standard dictionary sense ["Sentience is the ability to feel, perceive, or be conscious, or to experience subjectivity": http://en.wikipedia.org/wiki/Sentience ] Or are you using the term in some revisionary sense? At least if we discount radical philosophical scepticism about other minds, cows and other nonhuman vertebrates undergo phenomenal pain, anxiety, sadness, happiness and a whole bunch of phenomenal sensory experiences. For sure, cows are barely more sapient than a human prelinguistic toddler (though see e.g. http://www.appliedanimalbehaviour.com/article/S0168-1591(03)00294-6/abstract http://www.dailymail.co.uk/news/article-2006359/Moo-dini-Cow-unusual-intelligence-opens-farm-gate-tongue-herd-escape-shed.html ] But their limited capacity for abstract reasoning is a separate issue.
I suggest that to this array of terms, we should add moral indexicalism to designate Eliezer's position, which by the above definition would be a special form of realism. As far as I can tell, he basically says that moral terms are hidden indexicals in Putnam's sense.
And here we see the value of replacing the symbol with the substance.
This is something I've been meaning to ask about for a while. When humans say it is moral to satisfy preferences, they aren't saying that because they have an inbuilt preference for preference-satisfaction (or are they?). They're idealizing from their preferences for specific things (survival of friends and family, lack of pain, fun...) and making a claim that, ceteris paribus, satisfying preferences is good, regardless of what the preferences are.
Seen in this light, Clippy doesn't seem like quite as morally orthogonal to us as it once did. Clippy prefers paperclips, so ceteris paribus (unless it hurts us), it's good to just let it make paperclips. We can even imagine a scenario where it would be possible to "torture" Clippy (e.g., by burning paperclips), and again, I'm willing to pronounce that (again, ceteris paribus) wrong.
Maybe I am confused here...
Clippy is more of a Lovecraftian horror than a fellow sentient - where by "Lovecraftian" I mean to invoke Lovecraft's original intended sense of terrifying indifference - but if you want to suppose a Clippy that possesses a pleasure-pain architecture and is sentient and then sympathize with it, I suppose you could. The point is that your sympathy means that you're motivated by facts about what some other sentient being wants. This doesn't motivate Clippy even with respect to its own pleasure and pain. In the long run, it has decided, it's not out to feel happy, it's out to make paperclips.
Right, that makes sense. What interests me is (a) whether it is possible for Clippy to be properly motivated to make paperclips without some sort of phenomenology of pleasure and pain*, (b) whether human preference-for-preference-satisfaction is just another of many oddball human terminal values, or is arrived at by something more like a process of reason.
This is a difficult question, but I suppose that pleasure and pain are a mechanism for human (or other species') learning. Simply said: you do a random action, and the pleasure/pain response tells you it was good/bad, so you should make more/less of it again.
Clippy could use an architecture with a different model of learning. For example Solomonoff priors and Bayesian updating. In such architecture, pleasure and pain would not be necessary.
That leaves the sense in which you are not a moral realist most unclear.
That tacitly assumes that the question "does pleasure/happiness motivate posiively in all cases" is an emprical question -- that it would be possible to find an enitity that hates pleasure and loves pain. it could hover be plausibly argued that it is actually an analytical, definitional issue...that is some entity oves X and hates Y, we would just call X it's pleasure and Y its pain.
I suppose some non-arbitrary subjectivism is the obvious answer.
Are you trying to appeal to instrumental rationality, epistemic rationality, or some other kind?
An instrumentally rational agent wouldn't disregard a stronger preference of theirs in favour of a weaker preference of theirs. But other agent's preferences are not my own, and instrumental rationality does not oblige me to promote them.
Nor do epistemically rational agents have to abstain from meat. Even if moral realism were true, and eating animals were wrong, it doesn't follow that moral internalism is true: you might know that eating animals was wrong, but not be motivated to abstain. And even if knowledge did imply overwhelming motivation (an implausibly strong version of moral internalism) , our epistemically rational agent still wouldn't be obliged to abstain from eating meat, as she might simply be unaware of this moral truth. epistemic rationality doesn't imply omniscience.
If you have some other conception of rationality in mind, you have no objection to Eliezer, as he is only concerned with these two kinds.
pragmatist, apologies if I gave the impression that by "impartially gives weight" I meant impartially gives equal weight. Thus the preferences of a cow or a pig or a human trump the conflicting interests of a less sentient Anopheles mosquito or a locust every time. But on the conception of rational agency I'm canvassing, it is neither epistemically nor instrumentally rational for an ideal agent to disregard a stronger preference simply because that stronger preference is entertained by a member of a another species or ethnic group. Nor is it epistemically or instrumentally rational for an ideal agent to disregard a conflicting stronger preference simply because her comparatively weaker preference looms larger in her own imagination. So on this analysis, Jane is not doing what "an ideal agent (a perfectly rational agent, with infinite computing power, etc.) would choose."
Rationality can be used toward any goal, including goals that don't care about anyone's preference. For example, there's nothing in the math of utility maximisation that requires averaging over other agents' preferences (note: do not confuse utility maximisation with utilitarianism, they are very different things, the former being a decision theory, the latter being a specific moral philosophy).
nshepperd, utilitarianism conceived as theory of value is not always carefully distinguished from utilitarianism - especially rule-utilitarianism - conceived as a decision procedure. This distinction is nicely brought out in the BPhil thesis of FHI's Tony Ord, "Consequentialism and Decision Procedures": http://www.amirrorclear.net/academic/papers/decision-procedures.pdf Toby takes a global utilitarian consequentialist approach to the question, 'How should I decide what to do?" - a subtly different question from '"What should I do?"
This is potentially misleading. First, there's a good sense in which, moral realism or no moral realism, instrumental rationality does play a strong role in telling us what we 'should' want -- far more than does epistemic rationality. After all, instrumental rationality is often a matter of selecting which lower-level desires satisfy our higher-level ones, or selecting which desires in general form a coherent whole that is attainable from the present desire set.
But by 'what we should want' you don't mean 'what we should want in light of our other values'; you seem rather to mean 'what we should want in light of objective, unconditional moral facts'. (See my response to EY.) You're right that if there are such facts, then insofar as we can come to know them, epistemic rationality will direct us toward knowing them. But you haven't defended the independent assumption that knowing moral facts forces any rational agent to obey those facts.
Let's assume that moral realism (in the sense you seem to mean -- what I call moral unconditionalism) is true and, moreover, that the relevant facts are knowable. (Those are both very big assumptions, but I'm curious as to what follows.) How could we then argue that these facts are internalist, in the strong sense that they would be completely ineffable to any rational agent who was not thereby motivated to obey the dictates of these facts? In particular, how can we demonstrate this fact non-trivially, i.e., without building 'follows the instructions of any discovered moral facts' into our definition of 'rational'?
Does our concept of epistemic rationality, in itself, require that if agent A learns normative fact N (say, 'murder is wrong'), A must then eschew murder or else be guilty of e-irrationality? Clearly not. E-rationality is only about making your beliefs better fit the world. Further actions concerning those beliefs -- like following any instructions they contain, or putting them in alphabetical order, or learning the same facts in as many different languages as possible -- are extraneous to e-rationality.
(ETA: A perfectly e-rational agent is not even required to follow normative facts purely about beliefs that she learns, except insofar as following those norm-facts happens to foreseeably promote belief-accuracy. A purely e-rational agent who learns that murder is objectively wrong will not thereby be motivated to avoid murdering people, unless learning that murdering is wrong somehow leads the agent to conclude that murdering people will tend to make the agent acquire false beliefs or ignore true ones.)
Does our concept of instrumental rationality, in itself, require that if agent B learns normative fact N, B must then eschew murder or else be guilty of i-irrationality? Again, it's hard to see why. An agent is i-rational iff it tends to actualize situations it values. If B's values don't already include norms like 'follow any instructions you find embedded in moral facts', then there seems to be no inconsistency or (i-)irrationality in B's decision to disregard these facts in conduct, even if the agent is also completely e-rational and knows with certainty about these moral facts.
So in what sense is it (non-trivially) irrational to learn a moral fact, and then refuse to follow its dictate? What concept of rationality do you have in mind, and why should we care about the relevant concept?
Dave might not be explaining his own position as clearly as one might wish, but I think the core of his objection is that Jane is not being epistemically rational when she decides to eat other sentient beings. This is because she is acting on a preference rooted in a belief which doesn't adequately represent certain aspects of the world--specifically, the subjective points of view of sentient members of non-human animal species.
(BTW, I believe this subthread, including Eliezer's own helpful comments below, provides some evidence that Eliezer's policy of penalizing replies to comments below a certain karma threshold is misguided.
EDIT: David's comment above had -6 karma when I originally posted this reply. His current score is no longer below the threshold.)
Yes, although our conception of epistemic and instrumental rationality is certainly likely to influence our ethics, I was making a point about epistemic and instrumental rationality. Thus imagine if we lived in a era where utopian technology delivers a version of ubiquitous naturalised telepathy, so to speak. Granted such knowledge, for an agent to act in accordance with a weaker rather than stronger preference would be epistemically and instrumentally irrational. Of course, we don't (yet) live in a era of such radical transparency. But why should our current incomplete knowledge / ignorance make it instrumentally rational to fail to take into consideration what one recognises, intellectually at least, as the stronger preference? In this instance, the desire not to have one's throat slit is a very strong preference indeed.
["Replies to downvoted comments are discouraged. Pay 5 Karma points to proceed anyway?" says this Reply button. How bizarre. Is this invitation to groupthink epistemically rational? Or is killing cows good karma?]
Larks, we can of course stipulatively define "rational" so as to exclude impartial consideration of the preferences of other agents or subjects of experience. By this criterion, Jane is more rational than Jill - who scrupulously weighs the preferences of other subjects of experience before acting, not just her own, i.e. Jill aspires to a more inclusive sense of instrumental rationality. But why favour Jane's folk usage of "rational"? Jane's self-serving bias arbitrarily privileges one particular here-and-now over all other first-person perspectives. If the "view from nowhere" offered by modern science is correct, then Jane's sense she is somehow privileged or ontologically special is an illusion of perspective - genetically adaptive, for sure, but irrational. And false.
[No, this argument is unlikely to win karma with burger eaters :-)
David, we're not defining rationality to exclude other-oriented desires. We're just not including that exact morality into the word "rational". Instrumental rationality links up a utility function to a set of actions. You hand over a utility function over outcomes, epistemic rationality maps the world and then instrumental rationality hands back a set of actions whose expected score is highest. So long as it can build a well-calibrated, highly discriminative model of the world and then navigate to a compactly specified set of outcomes, we call it rational, even if the optimization target is "produce as many paperclips as possible". Adding a further constraint to the utility function that it be perfectly altruistic will greatly reduce the set of hypothetical agents we're talking about, but it doesn't change reality (obviously) nor yield any interesting changes in terms of how the agent investigates hypotheses, the fact that the agent will not fall prey to the sunk cost fallacy if it is rational, and so on. Perfectly altruistic rational agents will use mostly the same cognitive strategies as any other sort of rational agent, they'll just be optimizing for one particular thing.
Jane doesn't have any false epistemic beliefs about being special. She accurately models the world, and then accurately calculates and outputs "the strategy that leads to the highest expected number of burgers eaten by Jane" instead of "the strategy that has the highest expected fulfillment of all thinking beings' values".
Besides, everyone knows that truly rational entities only fulfill other beings' values if they can do so using friendship and ponies.
Eliezer, I'd beg to differ. Jane does not accurately model the world. Accurately modelling the world would entail grasping and impartially weighing all its first-person perspectives, not privileging a narrow subset. Perhaps we may imagine a superintelligent generalisation of http://www.guardian.co.uk/science/2013/feb/28/brains-rats-connected-share-information http://www.guardian.co.uk/science/brain-flapping/2013/mar/01/rats-are-like-the-borg With perfect knowledge of all the first-person facts, Jane could not disregard the strong preference of the cow not to be harmed. Of course, Jane is not capable of such God-like omniscience. No doubt in common usage, egocentric Jane displays merely a lack of altruism, not a cognitive deficit of reason. But this is precisely what's in question. Why build our canons of rational behaviour around a genetically adaptive delusion?
Accurately modeling the world entails making accurate predictions about it. An expected paperclip maximizer fully grasps the functioning of your brain and mind to the extent that this is relevant to producing paperclips; if it needs to know the secrets of your heart in order to persuade you, it knows them. If it needs to know why you write papers about the hard problem of conscious experience, it knows that too. The paperclip maximizer is not moved by grasping your first-person perspective, because although it has accurate knowledge of this fact, that is not the sort of fact that figures in its terminal values. The fact that it perfectly grasps the compellingness-to-Jane, even the reason why Jane finds certain facts to be inherently and mysteriously compelling, doesn't compel it. It's not a future paperclip.
I know exactly why the villain in Methods of Rationality wants to kill people. I could even write the villain writing about the ineffable compellingness of the urge to rid the world of certain people if I put that villain in a situation where he or she would actually read about the hard problem of conscious experience, and yet I am not likewise compelled. I don't have the perfect understanding of any particular real-world psychopath that I do of my fictional killer, but if I did know why they were killers, and of course brought to bear my standard knowledge of why humans write what they do about consciousness, I still wouldn't be compelled by even the limits of a full grasp of their reasons, their justifications, their inner experience, and the reasons they think their inner experience is ineffably compelling.
David, have you already read all this stuff on LW, in which case I shouldn't bother recapitulating it? http://lesswrong.com/lw/sy/sorting_pebbles_into_correct_heaps/, http://lesswrong.com/lw/ta/invisible_frameworks/, and so on?
For sure, accurately modelling the world entails making accurate predictions about it. These predictions include the third-person and first-person facts [what-it's-like-to-be-a-bat, etc]. What is far from clear - to me at any rate - is whether super-rational agents can share perfect knowledge of both the first-person and third-person facts and still disagree. This would be like two mirror-touch synaesthetes having a fist fight.
Thus I'm still struggling with, "The paperclip maximizer is not moved by grasping your first-person perspective." From this, I gather we're talking about a full-spectrum superintelligence well acquainted with both the formal and subjective properties of mind, insofar as they can be cleanly distinguished. Granted your example Eliezer, yes, if contemplating a cosmic paperclip-deficit causes the AGI superhuman anguish, then the hypothetical superintelligence is entitled to prioritise its super-anguish over mere human despair - despite the intuitively arbitrary value of paperclips. On this scenario, the paperclip-maximising superintelligence can represent human distress even more faithfully than a mirror-touch synaesthete; but its own hedonic range surpasses that of mere humans - and therefore takes precedence.
However, to be analogous to burger-choosing Jane in Luke's FAQ, we'd need to pick an example of a superintelligence who wholly understands both a cow's strong preference not to have her throat slit and Jane's comparatively weaker preference to eat her flesh in a burger. Unlike partially mind-blind Jane, the superintelligence can accurately represent and impartially weigh all relevant first-person perspectives. So the question is whether this richer perspective-taking capacity is consistent with the superintelligence discounting the stronger preference not to be harmed? Or would such human-like bias be irrational? In my view, this is not just a question of altruism but cognitive competence.
[Of course, given we're taking about posthuman superintelligence, the honest answer is boring and lame: I don't know. But if physicists want to know the "mind of God," we should want to know God's utility function, so to speak.]
Why not? Actions are a product of priors, perceptions and motives. Sharing perceptions isn't sharing motives - and even with identical motives, agents could still fight - if they were motivated to do so.
God's Utility Function according to Dawkins and Tyler.
Why not ?
Even if it turns out that all humans would become cow-compassionate given ultimate knowledge, we are still interested in the rationality of cow-satan.
Why not? Because Jane would weigh the preference of the cow not to have her throat slit as if it were her own. Of course, perfect knowledge of each other's first-person states is still a pipedream. But let's assume that in the future http://www.independent.co.uk/news/science/mindreading-rodents-scientists-show-telepathic-rats-can-communicate-using-braintobrain-8515259.html is ubiquitous, ensuring our mutual ignorance is cured.
"The rationality of cow satan"? Apologies Kyre, you've lost me here. Could you possibly elaborate?
What I'm saying is that cow-satan completely understands the preference of the cow not to have its throat slit. Every last grisly detail; all the physical, emotional, social, intellectual consequences, or consequences of any other kind. Cow satan has virtually experienced being slaughtered. Cow satan has studied the subject for centuries in detail. It is safe to say that no cow has ever understood the preference of cows not to be killed and eaten better than any cow ever could. Cow satan weighs that preference at zero.
It might be the case that cow satan could not actually exist in our universe, but would you say that it is irrational for him to go ahead and have the burger ?
(edit - thinking about it, that last question isn't perhaps very helpful)
Are you saying that perfect (or sufficiently good) mutual knowledge of each other's experiences would be highly likely to change everyone's preferences ? That might be the case, but I don't see how that makes Jane's burger choice irrational.
Yes Kyre, "Cow Satan", as far as I can tell, would be impossible. Imagine a full cognitive generalisation of http://www.livescience.com/1628-study-people-literally-feel-pain.html Why don't mirror-touch synaesthetes - or full-spectrum superintelligences - wantonly harm each other?
[this is not to discount the problem of Friendly AI. Alas one can imagine "narrow" superintelligences converting cows and humans alike into paperclips (or worse, dolorium) without insight into the first-person significance of what they are doing.]
There isn't too much that is impossible. In general, if we can imagine it, we can build it (because we have already built it - inside our brains).
See also:
That did not address David's True Rejection.
an Austere Charitable Metaethicist could do better.
The grandparent is a superb reply and gave exactly the information needed in a graceful and elegant manner.
Indeed it does. Not. Here is a condition in which I think David would be satified. If people would use vegetables for example as common courtesy to vegetarians, in the exact same sense that "she" has been largely adopted to combat natural drives towards "he"-ness. Note how Luke's agents and examples are overwhelmingly female. Not a requirement, just a courtesy.
An I don't say that as a vegetarian, because I'm not one.
Indeed. What is the Borg's version of the Decision Theory FAQ? This is not to say that rational agents should literally aim to emulate the Borg. Rather our conception of epistemic and instrumental rationality will improve if / when technology delivers ubiquitous access to each other's perspectives and preferences. And by "us" I mean inclusively all subjects of experience.
Hello, David.
If "What Do We Mean By 'Rationality'?" does not describe your conception of rationality, I am wondering, what is your conception of rationality? How would you define that term?
Given that, why should I care about being rational in your sense of the word? When I find ants, spiders, and other bugs in my house, I kill them. Sometimes I don't finish the job on the first try. I'm sure they are feeling pain, but I don't care. Sometimes, I even enjoy the feeling of "the hunt" and am quite satisfied when I'm done. Once, hornets built a large nest on my family's garage. We called an exterminator and had it destroyed. Again, I was quite happy with that decision and felt no remorse. Multiple times in my life, I have burned ants on my driveway to death with a magnifying glass, and, though I sometimes feel guilty about having done this, in the moment, I knew that the ants were suffering and actually enjoyed the burning, in part, for that very reason. The ants even squealed at the moment of their deaths, and that was my favorite part, again because it gave me the feeling of success in "the hunt".
No third-person fact you give me here will change my mind. You could rewire my brain so I felt empathy towards ants and other bugs, but I don't want you to do that. Unless I have misrepresented your conception of rationality, I think it fails to generally motivate (and there are probably many examples in which this occurs, besides mine).
Also, in case it comes up, I am a motivational externalist in the moral domain (though you probably have surmised that by now).
Maybe you need to talk to someone about it.
I don't burn ants anymore. My psychological health now is far superior to my psychological health back when I burned ants.
Have you considered immproving your psychological health so far you don't kill spiders, too?
Not sure if that was meant to be sarcastic, but I think it is fairly common for people to kill bugs that they find crawling around in their own home. Torturing them is a different matter.
I usually just ignore them, or if they bother me too much I try to get them out of the window alive.
Well, I don't, and I complain when I see people do it. But I'm atypical.
... how about not torturing and killing animals for food? Sure, most people do it, but most people are crazy.
I have much less of a problem with that (I eat meat myself, once in a while) than with torturing and killing animals for fun.
I actually went vegetarian last summer for a couple months. I survived, but I did not enjoy it. I definitely could not stand going vegan; I enjoy milk too much. When I went vegetarian, I did not feel nourished enough and I was unable to keep up my physique. At some meals, I couldn't eat with the rest of my family.
I would go vegetarian again if I had the finances to hire a personal trainer (who could guide me on how to properly nourish myself) and if I had the motivation to prepare many more meals for myself than I do right now. However, I don't, on both counts.
On the other hand, I did recently find out about something called Soylent, which I hope I will eventually be able to try out. Does that mesh better with your moral sensibilities? (honest question, not meant to sound edgy)
notsonewuser, a precondition of rational agency is the capacity accurately to represent the world. So in a sense, the local witch-doctor, a jihadi, and the Pope cannot act rationally - maybe "rational" relative to their conceptual scheme, but they are still essentially psychotic. Epistemic and instrumental rationality are intimately linked. Thus the growth of science has taken us all the way from a naive geocentrism to Everett's multiverse. Our idealised Decision Theory needs to reflect this progress. Unfortunately, trying to understand the nature of first-person facts and subjective agency within the conceptual framework of science is challenging, partly because there seems no place within an orthodox materialist ontology for the phenomenology of experience; but also because one has access only to an extraordinarily restricted set of first-person facts at any instant - the contents of a single here-and now. Within any given here-and-now, each of us seems to be the centre of the universe; the whole world is centred on one's body-image. Natural selection has designed us -and structured our perceptions - so one would probably lay down one's life for two of one's brothers or eight of one's cousins, just as kin-selection theory predicts; but one might well sacrifice a small third-world country rather than lose one's child. One's own child seems inherently more important than a faraway country of which one knows little. The egocentric illusion is hugely genetically adaptive. This distortion of perspective means we're also prone to massive temporal and spatial discounting. The question is whether some first-person facts are really special or ontologically privileged or deserve more weight simply because they are more epistemologically accessible? Or alternatively, is a constraint on ideal rational action that we de-bias ourselves?
Granted the scientific world picture, then, can it be rational to take pleasure in causing suffering to other subjects of experience just for the sake of it? After all, you're not a mirror-touch synaesthete. Watching primitive sentients squirm gives you pleasure. But this is my point. You aren't adequately representing the first-person perspectives in question. Representation is not all-or-nothing; representational fidelity is dimensional rather than categorical. Complete fidelity of representation entails perfectly capturing every element of both the formal third-person facts and subjective-first-person facts about the system in question. Currently, none of us yet enjoys noninferential access to other minds - though technology may shortly overcome our cognitive limitations here. I gather your neocortical representations sometimes tend causally to covary with squirming sentients. Presumably, their squirmings trigger the release of endogenous opioids in your hedonic hotspots, You enjoy the experience! (cf. http://news.nationalgeographic.com/news/2008/11/081107-bully-brain.html) But insofar as you find the first-person state of being panic-stricken as in any way enjoyable, you have misrepresented its nature. By analogy, a masochist might be turned on watching a video involving ritualised but nonconsensual pain and degradation. The co-release of endogenous opioids within his CNS prevents the masochist from adequately representing what's really happening from the first-person perspective of the victim. The opioids colour the masochist's representations with positive hedonic tone. Or to use another example, stimulate the relevant bit of neocortex with microelectrodes and you will find everything indiscriminately funny (cf.http://news.bbc.co.uk/1/hi/sci/tech/55893.stm ) - even your child drowning before your eyes. Why intervene if it's so funny? Although the funniness seems intrinsic to one's representations, they are misrepresentations to the extent they mischaracterise the first-person experiences of the subject in question. There isn't anything intrinsically funny about a suffering sentient. Rightly or wrongly, I assume that full-spectrum superintelligences will surpass humans in their capacity impartially to grasp first-person and third-person perspectives - a radical extension of the runaway mind-reading prowess that helped drive the evolution of disjunctively human intelligence.
So, no, without rewiring your brain, I doubt I can change your mind. But then if some touchy-feely superempathiser says they don't want to learn about quantum physics or Bayesian probability theory, you probably won't change their mind either. Such is life. If we aspire to be ideal rational agents - both epistemically and instrumentally rational - then we'll impartially weigh the first-person and third-person facts alike.
Hi David,
Thanks for your long reply and all of the writing you've done here on Less Wrong. I only hope you eventually see this.
I've thought more about the points you seem to be trying to make and find myself in at least partial agreement. In addition to your comment that I'm replying to, this comment you made also helped me understand your points better.
Just to clarify, you mean that human representation of others' pain is only represented using a (very) lossy compression, am I correct? So we end up making decisions without having all the information about those decisions we are making...in other words, if we computed the cow's brain circuitry within our own brains in enough detail to feel things the way they feel from the perspective of the cow, we obviously would choose not to harm the cow.
In at least one class of possible situations, I think you are definitely correct. If I were to say that my pleasure in burning ants outweighed the pain of the ants I burned (and thus that such an action was moral), but only because I do not (and cannot, currently) fully empathize with ants, then I agree that I would be making such a claim irrationally. However, suppose I already acknowledge that such an act is immoral (which I do), but still desire to perform it, and also have the choice to have my brain rewired so I can empathize with ants. In that case, I would choose not to have my brain rewired. Call this "irrational" if you'd like, but if that's what you mean by rationality, I don't see why I should be rational, unless that's what I already desired anyways.
The thing which you are calling rationality seems to have a lot more to do with what I (and perhaps many others on Less Wrong) would call morality. Is your sticking point on this whole issue really the word "rational", or is it actually on the word "ideal"? Perhaps burger-choosing Jane is not "ideal"; perhaps she has made an immoral choice.
How would you define the word "morality", and how does it differ from "rationality"? I am not at all trying to attack your position; I am trying to understand it better.
Also, I now plan on reading your work The Hedonistic Imperative. Do you still endorse it?
notsonewuser, yes, "a (very) lossy compression", that's a good way of putting it - not just burger-eating Jane's lossy representation of the first-person perspective of a cow, but also her lossy representation of her pensioner namesake with atherosclerosis forty years hence. Insofar as Jane is ideally rational, she will take pains to offset such lossiness before acting.
Ants? Yes, you could indeed choose not to have your brain reconfigured so as faithfully to access their subjective panic and distress. Likewise, a touchy-feely super-empathiser can choose not to have her brain reconfigured so she better understands of the formal, structural features of the world - or what it means to be a good Bayesian rationalist. But insofar as you aspire to be an ideal rational agent, then you must aspire to maximum representational fidelity to the first-person and the first-third facts alike. This is a constraint on idealised rationality, not a plea for us to be more moral - although yes, the ethical implications may turn out to be profound.
The Hedonistic Imperative? Well, I wrote HI in 1995. The Abolitionist Project (2007) (http://www.abolitionist.com) is shorter, more up-to-date, and (I hope) more readable. Of course, you don't need to buy into my quirky ideas on ideal rationality or ethics to believe that we should use biotech and infotech to phase out the biology of suffering throughout the living world.
On a different note, I don't know who'll be around in London next month. But on May 11, there is a book launch of the Springer volume, "Singularity Hypotheses: A Scientific and Philosophical Assessment":
http://www.meetup.com/London-Futurists/events/110562132/?a=co1.1_grp&rv=co1.1
I'll be making the case for imminent biologically-based superintelligence. I trust there will be speakers to put the Kurzweilian and MIRI / lesswrong perspective. I fear a consensus may prove elusive. But Springer have a commissioned a second volume - perhaps to tie up any loose ends.
What are you talking about? If you like utility functions, you don't argue about them (at least not on rationality grounds)! If I want to privilege this or that, I am not being irrational, I am at most possibly being a bastard.
IlyaShpitser, is someone who steals from their own pension fund an even bigger bastard, as you put it? Or irrational? What's at stake here is which preferences or interests to include in a utility function.
I don't follow you. What preferences I include is my business, not yours. You don't get to pass judgement on what is rational, rationality is just "accounting." We simply consult the math and check if the number is maximized. At most you can pass judgement on what is moral, but that is a complicated story.
IlyaShpitser, you might perhaps briefly want to glance through the above discussion for some context [But don't feel obliged; life is short!] The nature of rationality is a controversial topic in the philosophy of science (cf. http://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions). Let's just say if either epistemic or instrumental rationality were purely a question of maths, then the route to knowledge would be unimaginably easier.
Not necessarily if the math is really difficult. There are, after all, plenty of mathematical problems which have never been solved.
You are not going to ''do'' rationality unless you have a preference for it. And to have a preference for it is to have a preference for other things, like objectivity.
Look, I am not sure exactly what you are saying here, but I think you might be saying that you can't have Clippy. Clippy worries less about assigning weight to first and third person facts, and more about the fact that various atom configurations aren't yet paperclips. I think Clippy is certainly logically possible. Is Clippy irrational? He's optimizing what he cares about..
I think maybe there is some sort of weird "rationality virtue ethics" hiding in this series of responses.
Sure, it's only because appelatives like "bastard" imply a person with a constant identity through time that we call someone who steals from other people's pension funds a bastard, and from his own pension fund stupid or akratic. If we shrunk our view of identity to time-discrete agents making nanoeconomic transactions with future and past versions of themselves, we could call your premature pensioner a bastard; if we grew our view of identity to "all sentient beings," we could call someone who steals from others' pension funds stupid or akratic.
We could also call a left hand tossing a coin thrown by the right hand a thief; or divide up a single person into multiple, competing agents any number of other ways.
However, the choice of a assigning a consistent identity to each person is not arbitrary. It's fairly universal, and fairly well-motivated. Persons tend to be capable of replication, and capable of entering into enforceable contracts. Neither of the other agentic divisions--present/future self, left hand/right hand, or "all sentient beings"--share these characteristics. And these characteristics are vitally important, because agents that possess them can outcompete others that vie for the same resources; leaving the preferences of those other agents near-completely unsatisfied.
So, that's why LWers, with their pragmatic view toward rationality, aren't eager to embrace a definition of "rationality" that leaves its adherents in the dustbin of history unless everyone else embraces it at the same time.
Pragmatic? khafra, possibly I interpreted the FAQ too literally. ["Normative decision theory studies what an ideal agent (a perfectly rational agent, with infinite computing power, etc.) would choose."] Whether in practice a conception of rationality that privileges a class of weaker preferences over stronger preferences will stand the test of time is clearly speculative. But if we're discussing ideal, perfectly rational agents - or even crude approximations to ideal perfectly rational agents - then a compelling case can be made for an impartial and objective weighing of preferences instead.
You're sticking pretty determinedly to "preferences" as something that can be weighed without considering the agent that holds/implements them. But this is prima facie not how preferences work--this is what I mean by "pragmatic." If we imagine an ordering over agents by their ability to accomplish their goals, instead of by "rationality," it's clear that:
A preference held by no agents will only be satisfied by pure chance,
A preference held only by the weakest agent will only be satisfied if it is compatible with the preferences of the agents above it, and
By induction over the whole numbers, any agent's preferences will only be satisfied to the extent that they're compatible with the preferences of the agents above it.
As far as I can see, this leaves you with a trilemma:
There is no possible ordering over agents by ability to accomplish goals.
"Rationality" has negligible effect on ability to accomplish goals.
There exists some Omega-agent above all others, whose goals include fulfilling the preferences of weaker agents.
Branch 3 is theism. You seem to be aiming for a position in between branch 1 and branch 2; switching from one position to the other whenever someone attacks the weaknesses of your current position.
Edit: Whoops, also one more, which is the position you may actually hold:
4. Being above a certain, unspecified position in the ordering necessarily entails preferring the preferences of weaker agents. It's obvious that not every agent has this quality of preferring the preferences of weaker agents; and I can't see any mechanism whereby that preference for the preferences of weaker agents would be forced upon every agent above a certain position in the ordering except for the Omega-agent. So I think that mechanism is the specific thing you need to argue for, if this is actually your position.
Epistemic rationality may well affect what you should want. Epistemic rationality values objective facts above subjective whims. It also values consistency. It may well be that these valuations affect the rest of ones value system, ie it would be inconsistent for me to hold self-centered values when there is no objective reason for me to be more important than anyone else.
(Edited to make sense)
No, it doesn't, not by its most common definition.
Tim, conduct a straw poll of native English speakers or economists and you are almost certainly correct. But we're not ( I hope!) doing Oxford-style ordinary language philosophy; "Rationality" is a contested term. Any account of normative decision theory, "what an ideal agent (a perfectly rational agent, with infinite computing power, etc.) would choose", is likely to contain background assumptions that may be challenged. For what it's worth, I doubt a full-spectrum superintelligence would find it epistemically or instrumentally rational to disregard a stronger preference in favour of a weaker preference - any more than it would be epistemically or instrumentally rational for a human spit-brain patient arbitrarily to privilege the preferences of one cerebral hemisphere over the other. In my view, locking bias into our very definition of rationality would be a mistake.
I recommend putting any proposed definition on a web page and linking to it. Using words in an unorthodox way in the midst of conversation can invite misunderstanding.
I don't think it is practical to avoid some level of egocentricity. Distant regions suffer from signal delays, due to the light speed limit. Your knowledge of them is out of date, your influence there is weakened by delays in signal propagation. Locality in physics apparently weighs against the "god's eye view" that you advocate.
Tim, in practice, yes. But this is as true in physics as in normative decision theory. Consider the computational challenges faced by, say, a galactic-sized superintelligence spanning 100,000 odd light years and googols of quasi-classical Everett branches.
[yes, you're right about definitions - but I hadn't intended to set out a rival Decision Theory FAQ. As you've probably guessed, all that happened was my vegan blood pressure rose briefly a few days ago when I read burger-choosing Jane being treated as a paradigm of rational agency.]
That's what I mean. It kinda sounds as though you are arguing against physics.
Tim, on the contrary, I was arguing that in weighing how to act, the ideal rational agent should not invoke privileged reference frames. Egocentric Jane is not an ideal rational agent.
Embodied agents can't avoid "privileged reference frames", though. They are - to some degree - out of touch with events distant to them. The bigger the agent gets, the more this becomes an issue. It becomes technically challenging for Jane to take account of Jill's preferences when Jill is far away - ultimately because of locality in physics. Without a god, a "god's eye view" is not very realistic. It sounds as though your "ideal rational agent" can't be embodied.
Tim, an ideally rational embodied agent may prefer no suffering to exist outside her cosmological horizon; but she is not rationally constrained to take such suffering - or the notional preferences of sentients in other Hubble volumes - into consideration before acting. This is because nothing she does as an embodied agent will affect such beings. By contrast, the interests and preferences of local sentients fall within the scope of embodied agency. Jane must decide whether the vividness and immediacy of her preference for a burger, when compared to the stronger but dimly grasped preference of a terrified cow not to have her throat slit, disclose some deep ontological truth about the world or a mere epistemological limitation. If she's an ideal rational agent, she'll recognise the latter and act accordingly.
The issue isn't just about things beyond cosmological horizons. All distances are involved. I can help my neighbour more easily than I can help someone from half-way around the world. The distance involved entails expenses relating to sensory and motor signal propagation. For example, I can give my neighbour 10 bucks and be pretty sure that they will receive it.
Of course, there are also other, more important reasons why real agents don't respect the preferences of others. Egocentricity is caused more by evolution than by simple physics.
Lastly, I still don't think you can hope to use the term "rational" in this way. It sounds as though you're talking about some kind of supermorality to me. "Rationality" means something too different.
This sounds incompatible with consequentialism, for reasons pointed out by Nick Bostrom.
It would only be incompatible with consequentialism if the world contains infinite amounts of value. Moreover, all plausible moral theories include a consequentialist principle: non-consequentialist theories simply say that, in many cases, you are permitted or required not to act as that principle would require (because this would be too costly for you, would violate other people's rights, etc.). Bostrom's paper raises a problem for ethical theory generally, not for consequentialism specifically.
You get problems if you think it's even possible (p>0) that the world be canonically infinite, not merely if the world actually is infinite.
Bostrom's paper is excellent, and I recommend people read it. It is long, but only because it is exhaustive.