Steelmanning the Chinese Room Argument

cousin_it

(This post grew out of an old conversation with Wei Dai.)

Imagine a person sitting in a room, communicating with the outside world through a terminal. Further imagine that the person knows some secret fact (e.g. that the Moon landings were a hoax), but is absolutely committed to never revealing their knowledge of it in any way.

Can you, by observing the input-output behavior of the system, distinguish it from a person who doesn't know the secret, or knows some other secret instead?

Clearly the only reasonable answer is "no, not in general".

Now imagine a person in the same situation, claiming to possess some mental skill that's hard for you to verify (e.g. visualizing four-dimensional objects in their mind's eye). Can you, by observing the input-output behavior, distinguish it from someone who is lying about having the skill, but has a good grasp of four-dimensional math otherwise?

Again, clearly, the only reasonable answer is "not in general".

Now imagine a sealed box that behaves exactly like a human, dutifully saying things like "I'm conscious", "I experience red" and so on. Moreover, you know from trustworthy sources that the box was built by scanning a human brain, and then optimizing the resulting program to use less CPU and memory (preserving the same input-output behavior). Would you be willing to trust that the box is in fact conscious, and has the same internal experiences as the human brain it was created from?

A philosopher believing in computationalism would emphatically say yes. But considering the examples above, I would say I'm not sure! Not at all!

(This post grew out of an old conversation with Wei Dai.)

Can you, by observing the input-output behavior of the system, distinguish it from a person who doesn't know the secret, or knows some other secret instead?

Clearly the only reasonable answer is "no, not in general".

Again, clearly, the only reasonable answer is "not in general".

A philosopher believing in computationalism would emphatically say yes. But considering the examples above, I would say I'm not sure! Not at all!

Here's what I think happened.

You observed something interesting happening in your brain, you labeled it "consciousness".
You observed that other humans are similar to you both in structure and in behavior, so you deduced that the same interesting thing is is happening in their brains, and labeled the humans "conscious".
You observed that a rock is not similar to you in any way, deduced that the same interesting thing is not happening in it, and labeled it "not conscious".
Then you observed a robot, and you asked "is it conscious?". If you asked the full question - "are the things happening in a robot similar to the things happening in my brain" - it would be obvious that you won't get a yes/no answer. They're similar in some ways and different in others.

But if you go back to the original question, you can't rule out that the robot is fully conscious , despite having some physical differences. The point being that translating questions about consciousness into questions about brain activity and function (in a wholesale and unguided way) isn't superior, it's potentially misleading.

0g_pepper9y

Yes, that sounds about right, with the caveat that I would say that other humans are almost certainly conscious. Obviously there are people (e.g. solipsists) who don't think that conscious minds other than their own exist. [...] That sounds approximately right, albeit it is not just the fact that a rock is dissimilar to me that leads me to believe it to be unconscious. I am open to the possibility that entities very different from myself might be conscious. [...] I'm not sure that "is the robot conscious" is really equivalent to "are the things happening in a robot similar to the things happening in my brain". It could be that some things happening in the robot's brain are similar in some ways to some things happening in my brain, but the specific things that are similar might have little or nothing to do with consciousness. Moreover, even if a robot's brain used mechanisms that are very different from those used by my own brain, this would not mean that the robot is necessarily not conscious. That is what makes the consciousness question difficult - we don't have an objective way of detecting it in others, particularly in others whose physiology differs significantly from our own. Note that this does not make consciousness unreal, however. I would be willing to answer "no" to the "is the robot conscious" question for any current robot that I have seen or even read about. But, that is not to say that no robot will ever be conscious.I do agree that there could be varying degrees of consciousness (rather than a yes/no answer), e.g. I suspect that animals have varying degrees of consciousness, e.g. non-human apes a fairly high degree, ants a low or zero degree, etc. I don't see why any of this would lead to the conclusion that consciousness or pain are not real phenomena.

10

Steelmanning the Chinese Room Argument

10

10

10

Steelmanning the Chinese Room Argument

10

10