I don't know how we could possibly know if the AI is aping consciousness for it own ends or if it actually is conscious.
How do you answer that question for human beings? ;-)
How do you know if a tree falling in the forest is just aping a sound for its own ends, or if it actually makes a sound? ;-)
Thinking about p-zombies is a sure sign that you are confused, and projecting a property of your brain onto the outside world. If you think there can be a p-zombie, then the mysterious thing called "consciousness" must reside in your map, not in the territory.
More specifically, if you can imagine robot or person A "with" consciousness, and robot or person B "without" it, and meanwhile imagine them to be otherwise "identical", then the only place for the representation of the idea that there is a difference, is in the mental tags you label these imaginary robots or people with.
This is why certain ideas seem "supernatural" or "beyond the physical" -- they are actually concepts that exist only in the mind of the observer, rather than in the physical reality.
This isn't to say that there might someday be a reductionist definition of consciousness that refers to facts in the territory, just that the intuitive idea of consciousness is actually a bit that gets flipped in our brains: an inbuilt bit that's part of our brain's machinery for telling the difference between animals, plants, and inanimate objects. This bit is the true home of our intuitive idea of consciousness, and is largely unrelated to whatever actual consciousness is.
In young children, after all, this bit flips on for anything that appears to move or speak by itself: animals, puppets, cartoons, etc. We can learn to switch it on or off for specific things, of course, but it's an inbuilt function that has its own sense or feel of being "real".
It's also why we can imagine the same thing having or not-having consciousness, because to our brain, it is an independent variable, because we have separate wiring to track that variable independently. But the mere existence of a tracking variable in our heads, doesn't mean the thing it tracks has a real existence. (We can see "colors" that don't really exist in the visible spectrum, after all: many colors are simply our perception of what happens when something reflects light at more than one wavelength.)
I suggest that if you are still confused about this, consult the sequence on the meaning of words, and particularly the bits about "How an algorithm feels from the inside" -- if I recall correctly, that's the bit that shows how our neural nets can represent conceptual properties like "makes a sound" as if they were independent of the physical phenomena involved. The exact same concept applies to our projections of consciousness, and the confusion of p-zombies.
How do you answer that question for human beings? ;-)
I don't but I assume that they do, because other humans and I share a common origin. I know I am conscious, and I would consider it strange if natural selection made me conscious, but not individuals that are basically genetically identical to me.
I suggest that if you are still confused about this, consult the sequence on the meaning of words, and particularly the bits about "How an algorithm feels from the inside"
I will review that sequence again. I certainly am confused. However, I don...
In response to the classic Mysterious Answers to Mysterious Questions, I express some skepticism that consciousness is can be understood by science. I postulate (with low confidence) that consciousness is “inherently mysterious”, in that it is philosophically and scientifically impenetrable. The mysteriousness is a fact about our state of mind, but that state of mind is due to a fundamental epistemic feature of consciousness and is impossible to resolve.
My issue with understanding the cause of consciousness involves p-zombies. Any experiment with the goal of understanding consciousness would have to be able to detect consciousness, which seems to me to be philosophically impossible. To be more specific, any scientific investigation of the cause of consciousness would have (to simplify) an independent variable that we could manipulate to see how this manipulation affects the dependent variable, the presence or absence of consciousness. We assume that those around us are conscious, and we have good reason to do so, but we can't rely on that assumption in any experiment in which we are investigating consciousness. Before we ask “what is causing x?”, we first have to know that x is present.
As Eliezer points out, that an individual says he's conscious is a pretty good signal of consciousness, but we can't necessarily rely on that signal for non-human minds. A conscious AI may never talk about its internal states depending on its structure. (Humans have a survival advantage to social sharing of internal realities; an AI will not be subject to that selection pressure. There’s no reason for it to have any sort of emotional need to share its feelings, for example.) On the flip side, a savvy but non-conscious AI may talk about it's "internal states", not because it actually has internal states, but because it is “guessing the teacher's password” in the strongest way imaginable: it has no understanding whatsoever of what those states are, but computes that aping internal states will accomplish it's goals. I don't know how we could possibly know if the AI is aping consciousness for it own ends or if it actually is conscious. If consciousness is thus undetectable, I can't see how science can investigate it.
That said, I am very well aware that “Throughout history, every mystery, ever solved has turned out to be not magic*” and that every single time something has seemed inscrutable to science, a reductionist explanation eventually surfaced. Knowing this, I seriously downgrade my confidence that "No, really, this time it is different. This phenomenon really is beyond the grasp of science." I look forward to someone coming forward with something clever that dissolves the question, but even so, it does seem inscrutable.
*- Though, to be fair, this is a selection bias. Of course, all the solved mysteries weren't magic. All the mysteries that are acctully magic remain unsolved, because they're magic! This is NOT to say I believe in magic, just to say that it's hardly saying much to claim that all the things we've come to understand were in principle understandable. To steelman: I do understand that with each mystery that was once declared to be magical, then later shown not to be, our collective priors for the existence of magical things decrease. (There is a sort of halting problem: if a question has remained unsolved since the dawn of asking questions, is that because it is unsolvable, or because we're right around the corner form solving it?)