But saying that e.g. rats are not sentient in the context of concern about the treatment of sentient beings is like saying that Negroes are not men in the context of the Declaration of Independence. Not only are the purely semantic aspects dubious, but excluding entities from a moral category on semantic grounds seems like a severe mistake regardless.
Words like "sentience" and especially "consciousness" are often used to refer to the soul without sounding dogmatic about it. You can tell this from the ways people use them: "Would a perfect duplicate of you have the same consciousness?", "Are chimps conscious?", etc. You can even use such terminology in such ways if you're a materialist who denies the existence of souls. You'd sound crazy talking about souls like they're real things if you say that there are no such things as souls, wouldn't you? Besides, souls are supernatural. Consciousness, on the other hand, is an emergent phenomenon, which sounds much more scientific.
Is there good reason to think that there is some sort of psychic élan vital? It strikes me as probably being about as real as phlogiston or luminiferous aether; i.e. you can describe phenomena in terms of the concept, and it doesn't necessarily prevent you from doing so basically correctly, but you can do better without it.
And, of course, in the no-nonsense senses of the terms, rats are sentient, conscious, aware, or however else you want to put it. Not all of the time, of course. They can also be asleep or dead or other things, as can humans, but rats are often sentient. And it's not hard to tell that plenty of non-humans also experience mental phenomena, which is why it's common knowledge that they do.
I can't recall ever seeing an argument that mistreating minds without self-awareness or metacognition or whatever specific mental faculty is arbitrarily singled out, is kinder or more just or in any normal sense more moral than mistreating a mind without it. And you can treat any position as a self-justifying axiom, so doing so doesn't work out to an argument for the position's truth in anything but a purely relativist sense.
It is both weird and alarming to see Eliezer arguing against blindly assuming that a mind is too simple to be "sentient" while also pretty clearly taking the position that anything much simpler than our own minds isn't. It really seems like he rather plainly isn't following his own advice, and that that could happen without him realizing it is very worrying. He has admitted that this is something he's confused about and is aware that others are more inclusive, but that doesn't seem to have prompted him to rethink his position all that much, which suggests that Eliezer is really confused about this in a way that may be hard to correct.
Looking for a nonperson predicate is kind of seeking an answer to the question "Who is it okay to do evil things to?" I would like to suggest that the correct answer is "No one", and that asking the question in the first place is a sign that you made a big mistake somewhere if you're trying to avoid being evil.
If having the right not to have something done to you just means that it's morally wrong to do that thing to you, then everything has rights. Making a rock suffer against it will would be, if anything, particularly evil, as it would require you to go out of your way to give the rock a will and the capacity to suffer. Obviously, avoiding violating anything's rights requires an ability to recognize what something wills, what will cause it to suffer, and/or etc. Those are useful distinctions. But it seems like Eliezer is talking about something different.
Has he written anything more recently on this subject?
Followup to: Righting a Wrong Question, Zombies! Zombies?, A Premature Word on AI, On Doing the Impossible
There is a subproblem of Friendly AI which is so scary that I usually don't talk about it, because very few would-be AI designers would react to it appropriately—that is, by saying, "Wow, that does sound like an interesting problem", instead of finding one of many subtle ways to scream and run away.
This is the problem that if you create an AI and tell it to model the world around it, it may form models of people that are people themselves. Not necessarily the same person, but people nonetheless.
If you look up at the night sky, and see the tiny dots of light that move over days and weeks—planētoi, the Greeks called them, "wanderers"—and you try to predict the movements of those planet-dots as best you can...
Historically, humans went through a journey as long and as wandering as the planets themselves, to find an accurate model. In the beginning, the models were things of cycles and epicycles, not much resembling the true Solar System.
But eventually we found laws of gravity, and finally built models—even if they were just on paper—that were extremely accurate so that Neptune could be deduced by looking at the unexplained perturbation of Uranus from its expected orbit. This required moment-by-moment modeling of where a simplified version of Uranus would be, and the other known planets. Simulation, not just abstraction. Prediction through simplified-yet-still-detailed pointwise similarity.
Suppose you have an AI that is around human beings. And like any Bayesian trying to explain its enivornment, the AI goes in quest of highly accurate models that predict what it sees of humans.
Models that predict/explain why people do the things they do, say the things they say, want the things they want, think the things they think, and even why people talk about "the mystery of subjective experience".
The model that most precisely predicts these facts, may well be a 'simulation' detailed enough to be a person in its own right.
A highly detailed model of me, may not be me. But it will, at least, be a model which (for purposes of prediction via similarity) thinks itself to be Eliezer Yudkowsky. It will be a model that, when cranked to find my behavior if asked "Who are you and are you conscious?", says "I am Eliezer Yudkowsky and I seem have subjective experiences" for much the same reason I do.
If that doesn't worry you, (re)read "Zombies! Zombies?".
It seems likely (though not certain) that this happens automatically, whenever a mind of sufficient power to find the right answer, and not otherwise disinclined to create a sentient being trapped within itself, tries to model a human as accurately as possible.
Now you could wave your hands and say, "Oh, by the time the AI is smart enough to do that, it will be smart enough not to". (This is, in general, a phrase useful in running away from Friendly AI problems.) But do you know this for a fact?
When dealing with things that confuse you, it is wise to widen your confidence intervals. Is a human mind the simplest possible mind that can be sentient? What if, in the course of trying to model its own programmers, a relatively younger AI manages to create a sentient simulation trapped within itself? How soon do you have to start worrying? Ask yourself that fundamental question, "What do I think I know, and how do I think I know it?"
You could wave your hands and say, "Oh, it's more important to get the job done quickly, then to worry about such relatively minor problems; the end justifies the means. Why, look at all these problems the Earth has right now..." (This is also a general way of running from Friendly AI problems.)
But we may consider and discard many hypotheses in the course of finding the truth, and we are but slow humans. What if an AI creates millions, billions, trillions of alternative hypotheses, models that are actually people, who die when they are disproven?
If you accidentally kill a few trillion people, or permit them to be killed—you could say that the weight of the Future outweighs this evil, perhaps. But the absolute weight of the sin would not be light. If you would balk at killing a million people with a nuclear weapon, you should balk at this.
You could wave your hands and say, "The model will contain abstractions over various uncertainties within it, and this will prevent it from being conscious even though it produces well-calibrated probability distributions over what you will say when you are asked to talk about consciousness." To which I can only reply, "That would be very convenient if it were true, but how the hell do you know that?" An element of a model marked 'abstract' is still there as a computational token, and the interacting causal system may still be sentient.
For these purposes, we do not, in principle, need to crack the entire Hard Problem of Consciousness—the confusion that we name "subjective experience". We only need to understand enough of it to know when a process is not conscious, not a person, not something deserving of the rights of citizenship. In practice, I suspect you can't halfway stop being confused—but in theory, half would be enough.
We need a nonperson predicate—a predicate that returns 1 for anything that is a person, and can return 0 or 1 for anything that is not a person. This is a "nonperson predicate" because if it returns 0, then you know that something is definitely not a person.
You can have more than one such predicate, and if any of them returns 0, you're ok. It just had better never return 0 on anything that is a person, however many nonpeople it returns 1 on.
We can even hope that the vast majority of models the AI needs, will be swiftly and trivially approved by a predicate that quickly answers 0. And that the AI would only need to resort to more specific predicates in case of modeling actual people.
With a good toolbox of nonperson predicates in hand, we could exclude all "model citizens"—all beliefs that are themselves people—from the set of hypotheses our Bayesian AI may invent to try to model its person-containing environment.
Does that sound odd? Well, one has to handle the problem somehow. I am open to better ideas, though I will be a bit skeptical about any suggestions for how to proceed that let us cleverly avoid solving the damn mystery.
So do I have a nonperson predicate? No. At least, no nontrivial ones.
This is a challenge that I have not even tried to talk about, with those folk who think themselves ready to challenge the problem of true AI. For they seem to have the standard reflex of running away from difficult problems, and are challenging AI only because they think their amazing insight has already solved it. Just mentioning the problem of Friendly AI by itself, or of precision-grade AI design, is enough to send them fleeing into the night, screaming "It's too hard! It can't be done!" If I tried to explain that their job duties might impinge upon the sacred, mysterious, holy Problem of Subjective Experience—
—I'd actually expect to get blank stares, mostly, followed by some instantaneous dismissal which requires no further effort on their part. I'm not sure of what the exact dismissal would be—maybe, "Oh, none of the hypotheses my AI considers, could possibly be a person?" I don't know; I haven't bothered trying. But it has to be a dismissal which rules out all possibility of their having to actually solve the damn problem, because most of them would think that they are smart enough to build an AI—indeed, smart enough to have already solved the key part of the problem—but not smart enough to solve the Mystery of Consciousness, which still looks scary to them.
Even if they thought of trying to solve it, they would be afraid of admitting they were trying to solve it. Most of these people cling to the shreds of their modesty, trying at one and the same time to have solved the AI problem while still being humble ordinary blokes. (There's a grain of truth to that, but at the same time: who the hell do they think they're kidding?) They know without words that their audience sees the Mystery of Consciousness as a sacred untouchable problem, reserved for some future superbeing. They don't want people to think that they're claiming an Einsteinian aura of destiny by trying to solve the problem. So it is easier to dismiss the problem, and not believe a proposition that would be uncomfortable to explain.
Build an AI? Sure! Make it Friendly? Now that you point it out, sure! But trying to come up with a "nonperson predicate"? That's just way above the difficulty level they signed up to handle.
But a blank map does not correspond to a blank territory. Impossible confusing questions correspond to places where your own thoughts are tangled, not to places where the environment itself contains magic. Even difficult problems do not require an aura of destiny to solve. And the first step to solving one is not running away from the problem like a frightened rabbit, but instead sticking long enough to learn something.
So let us not run away from this problem. I doubt it is even difficult in any absolute sense, just a place where my brain is tangled. I suspect, based on some prior experience with similar challenges, that you can't really be good enough to build a Friendly AI, and still be tangled up in your own brain like that. So it is not necessarily any new effort—over and above that required generally to build a mind while knowing exactly what you are about.
But in any case, I am not screaming and running away from the problem. And I hope that you, dear longtime reader, will not faint at the audacity of my trying to solve it.