= 27d567bc2d48d9f40e9ccc20ea370b15)
This essay is... unclear, but it really sounds like you are limiting the definition of 'intelligence' to a large but limited set of somewhat human-like intelligences with a native capacity for sociability, which does not include most Yudkowskian FOOMing AIs.
In response to
Arguments against the Orthogonality Thesis
I should have explained things much more at length. The intelligence in that context I use is general superintelligence, being defined as that which surpasses human intelligence in all domains. Why is a native capacity for sociability implied?
But how do they know what a Gods-eye view even is?
A "God's-eye view", as David Pearce says, is an impersonal view, an objective rather than subjective view, a view that does not privilege one personal perspective over another, but take the universe as a whole as its point of reference. This comes from the argued non-existence of personal identities. To check arguments on this, see this comment.
In response to
Arguments against the Orthogonality Thesis
Different instances in time of a physical organism relate to it in the same way that any other physical organism in the universe does. There is no logical basis for privileging a physical organism's own viewpoint,
Indeed, there is none. But nor is there any logical basis for not privileging a physical organism's own viewpoint, and since most organisms evolve/are built to privilege themselves, this is not an argument that will make them change their opinion.
In practical terms, it's very hard to change the intuitive opinions of people on this, even after many philosophical arguments. Those statements of mine don't touch the subject. For that the literature should be read, for instance the essay I wrote about it. But if we consider general superintelligences, then they could easily understand it and put it coherently into practice. It seems that this can be naturally expected, except perhaps in practice under some specific cases of human intervention.
While the physical world has no ethical value except from conscious perceptions, conscious perceptions can be ethical value, and only by being good or bad conscious perceptions, or feelings. This seems to be so by definition, because ethical value is being good or bad.
That a conscious experience can be a good or bad physical occurrence is also a reality which can be felt and known with the highest possible certainty.
A bit unclear, but I'm assuming you mean something like "we have good or bad (technically, pleasant or unpleasant) conscious experiences, and we know this with great certainty". That seems fine.
This makes it rational, and an imperative, to follow it and care about it, to act in order to foster good conscious feelings and to prevent bad conscious feelings, because it is logical that this will make the universe better.
Why? This is the whole core of the disagreement, and you're zooming over it way too fast. Even for ourselves, our wanting systems and our liking systems are not well aligned - we want things we don't like, and vice-versa. A preference utilitarian would say our wants are the most important; you seem to disagree, focusing on the good/bad aspect instead. But what logical reason would there be to follow one or the other?
You seem to get words to do too much of the work. We have innate senses of positivity and negativity for certain experiences; we also have an innate sense that morality exists. But those together do not make positive experiences good "by definition" (nor does calling them "good" rather than "positive").
But those are relatively minor points - if there was a single consciousness in the universe, them maybe your argument could get off the ground. But we have many current and potential consciousnesses, with competing values and conscious experiences. You seem to be saying that we should logically be altruists, because we have conscious experiences. I agree we should be altruists; but that's a personal preference, and there's no logic to it. Following your argument (consciousness before physics) one could perfectly become a solipsist, believing only one's own mind exists, and ignoring others. Or your could be a racist altruist, preferring certain individuals or conscious experiences. Or you could put all experiences together on an infinite numbers of comparative scales (there is no intrinsic measure to compare the quality of two positive experiences in different people).
But in a way, that's entirely a moot point. Your claim is that a certain ethics logically follows from our conscious reality. There I must ask you to prove it. State your assumptions, show your claims, present the deductions. You'll need to do that, before we can start critiquing your position properly.
Hi Stuart,
Why? This is the whole core of the disagreement, and you're zooming over it way too fast. Even for ourselves, our wanting systems and our liking systems are not well aligned - we want things we don't like, and vice-versa. A preference utilitarian would say our wants are the most important; you seem to disagree, focusing on the good/bad aspect instead. But what logical reason would there be to follow one or the other?
Indeed, wanting and liking do not always correspond, also from a neurological perspective. Wanting involves planning and planning often involves error. We often want things mistakenly, be it by evolutionary selected reasons, cultural reasons, or just bad planning. Liking is what matters, because it can be immediately and directly determined to be good, with the highest certainty. This is an empirical confirmation of its value, while wanting is like an empty promise.
We have good and bad feelings associated with some evolutionarily or culturally determined things. Theoretically, the result of good and bad feelings could be associated with any inputs. The inputs don't matter, nor does wanting necessarily matter, nor innate intuitions of morality. The only thing that has direct value, which is empirically confirmed, is good and bad feelings.
if there was a single consciousness in the universe, them maybe your argument could get off the ground. But we have many current and potential consciousnesses, with competing values and conscious experiences.
Well noticed. That comment was not well elaborated and is not a complete explanation. It is also necessary for that point you mentioned to consider the philosophy of personal identities, which is a point that I examine in my more complete essay on Less Wrong, and also in my essay Universal Identity.
But in a way, that's entirely a moot point. Your claim is that a certain ethics logically follows from our conscious reality. There I must ask you to prove it. State your assumptions, show your claims, present the deductions. You'll need to do that, before we can start critiquing your position properly.
I have a small essay written on ethics, but it's a detailed topic, and my article may be too concise, assuming much previous reading on the subject. It is here. I propose that we instead focus on questions as they come up.
In response to
Arguments against the Orthogonality Thesis
What sorts of actions, both cognitive (inside the computer) and physical, would a robot have to take to make you think it valued some alien thing like maximizing the number of paperclips in the world? Robbing humans to build a paperclip factory, for example, or making a list of plans, ordered by how many paperclips they would make.
Why is it impossible to program a robot that would do this?
For example, "there is no logical basis for privileging a physical organism's own viewpoint" is true under certain premises, but what goes wrong if we just build a robot that uses its own viewpoint anyways? Which parts of the program fail when it tries to selfishly maximize the number of paperclips?
Or if I were to make a suggestion rather than asking questions: Needs less cognitive language.
Indeed, a robot could be built that makes paperclips or pretty much anything. For instance, a paperclip assembling machine. That's an issue of practical implementation and not what the essay has been about, as I mention in the first paragraph and concede in the last.
The issue I argued about is that generally superintelligent agents, on their own will, without certain outside pressures from non-superintelligent agents, would understand personal identity and meta-ethics, leading them to converge to the same values and ethics. This is for two reasons: (1) they would need to take a "God's eye view" and value all perspectives besides their own, and (2) they would settle on moral realism, with the same values as good and bad feelings, in the present or future.
I read that and similar articles. I deliberately didn't say pleasure or happiness, but "reduced to good and bad feelings", including other feelings that might be deemed good, such as love, curiosity, self-esteem, meaningfulness..., and including the present and the future. The part about the future includes any instrumental actions in the present which be taken with the intention of obtaining good feelings in the future, for oneself or for others.
This should cover visiting Costa Rica, having good sex, and helping loved ones succeed, which are the examples given in that essay against the simple example of Nozick's experience machine. The experience machine is intuitively deemed bad because it precludes acting in order to instrumentally increase good feelings in the future and prevent bad feelings of oneself or others, and because pleasure is not what good feelings are all about. It is a very narrow part of the whole spectrum of good experiences one can have, precluding many others mentioned, and this makes it aversive.
The part about wanting and liking has neurological interest and has been well researched. It is not relevant for this question, because values need not correspond with wanting, they can just correspond with liking. Immediate liking is value, wanting is often mistaken. We want things which are evolutionarily or culturally caused, but that are not good for us. Wanting is like an empty promise, while liking can be empirically and directly verified to be good.
Any valid values reduce to good and bad feelings, for oneself or for others, in the present or in the future. This can be said of survival, learning, working, loving, protecting, sight-seeing, etc.
I say it again, I dare Eliezer (or others) to defend and justify a value that cannot be reduced to good and bad feelings.
In response to
Arguments against the Orthogonality Thesis
The crux of the disagreement, I think, is in the way we understand the self-assessment of our experience. If consciousness is epiphenomenal or just a different level of description of a purely physical world, this self-assessment is entirely algorithmic and does not disclose anything real about the intrinsic nature of consciousness.
But consciousness is not epiphenomenal, and a purely computational account fails to bridge the explanatory gap. Somehow conscious experience can evaluate itself directly, which still remains a not well understood and peculiar fact about the universe. In addition, as I see it, this needs to be acknowledged to make more progress in understanding both ethics and the relationship between the physical world and consciousness.
Indeed, epiphenomenalism can seemingly be easily disproved by its implication that if it were true, then we wouldn't be able to talk about our consciousness. As I said in the essay, though, consciousness is that of which we can be most certain of, by its directly accessible nature, and I would rather think that we are living in a virtual world under an universe with other, alien physical laws, than that consciousness itself is not real.
And it wouldn't defeat the OT because you'd still have to prove you couldn't have a utility function over e.g. causal continuity (note: you can have a utility function over causal continuity).
A certain machine could perhaps be programmed with an utility function over causal continuity, but a privileged stance for one's own values wouldn't be rational lacking a personal identity, in an objective "God's eye view", as David Pearce says. That would call at least for something like coherent extrapolated volition, at least including agents with contextually equivalent reasoning capacity. Note that I use "at least" twice, to accommodate your ethical views. More sensible would be to include not only humans, but all known sentient perspectives, because the ethical value(s) of subjects arguably depend more on sentience than on reasoning capacity.
In response to
Arguments against the Orthogonality Thesis
I actually read all the way through and found the broad argument quite understandable (although many of the smaller details were confusing). I also found it obviously wrong on many levels. The one I would consider most essential is that you say:
The existence of personal identities is purely an illusion that cannot be justified by argument, and clearly disintegrates upon deeper analysis...
Different instances in time of a physical organism relate to it in the same way that any other physical organism in the universe does. There is no logical basis for privileging a physical organism's own viewpoint, nor the satisfaction of their own values over that of other physical organisms...
Assuming I understood correctly, you're saying that because continuous personal identity isn't a real thing, there's no reason to favour one conscious being over another. But that doesn't follow at all. Just because the "you" a year from now is only somewhat similar to the "you" now doesn't mean you shouldn't favour him over everyone else (and indeed there are good reasons for doing so). I wrote a longer comment along these lines in response to some doubts in a recent a discussion of a post about dissolving personal identity.
I argue (in this article) that the you (consciousness) in one second bears little resemblance to the you in the next second.
In the subatomic world, the smallest passage of time changes our composition and arrangement to a great degree, instantly. In physical terms, the frequency of discrete change at this level, even in just one second, is a number with 44 digits, so vast as to be unimaginable... In comparison, the amount of seconds that have passed since the start of the universe, estimated at 13.5 billion years ago, is a number with just 18 digits. At the most fundamental level, our structure, which seems outwardly stable, moves at a staggering pace, like furiously boiling water. Many of our particles are continually lost, and new ones are acquired, as blood frantically keeps matter flowing in and out of each cell.
I also explain why you can't have partial identity in that paper, and that argues against the position you took (which is similar to that explained by philosopher David Lewis in his paper Survival and Identity).
If we were to be defined as a precise set of particles or arrangement thereof, its permanence in time would be implausibly short-lived; we would be born dead. If this were one's personal identity, it would have been set for the first time to one's baby state, having one's first sparkle of consciousness. In a subatomic level, each second is like many trillions of years in the macroscopic world, and our primordial state as a babies would be incredibly short-lived. In the blink of an eye, our similarity to what that personal identity was would be reduced to a tiny fraction, if any, by the sheer magnitude of change. That we could survive, in a sense, as a tiny fraction of what we once were, would be an hypothesis that goes against our experience, because we feel consciousness always as an integrated whole, not as a vanishing separated fraction. We either exist completely or not at all.
I recommend reading, whether you agree with this essay or not. The advanced and tenable philosophical positions on this subject are two. Empty individualism, characterized by Derek Parfit in his book "Reasons and Persons", and open individualism, for which there are better arguments, explained in 4 pages in my essay and more at length in Daniel Kolak's book "I Am You: The Metaphysical Foundations for Global Ethics".
For another interesting take on the subject here on Less Wrong, check Kaj Sotala's An attempt to dissolve subjective expectation and personal identity.
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
That is your opinion. Others believe wanting is fundamental and rational, that can be checked and explained and shared - while liking is a misleading emotional response (that probably shows much less consistency, too).
How would you resolve the difference? They say something is more important, you say something else is. Neither of your disagree about the facts of the world, just about what is important and what isn't. What can you point to that makes this into a logical disagreement?
One argument is that from empiricism or verification. Wanting can be and often is wrong. Simple examples can show this, but I assume that they won't be needed because you understand. Liking can be misleading in terms of motivation or in terms of the external object which is liked, but it cannot be misleading or wrong in itself, in that it is a good feeling. For instance, a person could like to use cocaine, and this might be misleading in terms of being a wrong motivation, that in the long-term would prove destructive and dislikeable. However, immediately, in terms of the sensation of liking itself, and all else being equal, then it is certainly good, and this is directly verifiable by consciousness.
Taking this into account, some would argue for wanting values X, Y, or Z, but not values A, B, or C. This is another matter. I'm arguing that good and bad feelings are the direct values that have validity and should be wanted. Other valid values are those that are instrumentally reducible to these, which are very many, and most of what we do.