Previously: Seeing Red: Dissolving Mary's Room and Qualia, A Study of Scarlet: The Conscious Mental Graph
When we left off, we'd introduced a hypothetical organism called Martha whose actions are directed by a mobile graph of simple mental agents. The tip of the iceberg, consisting of the agents that are connected to Martha's language centers, we called the conscious subgraph. Now we're going to place Martha into a situation like Mary's Room: we'll say that a large unconscious agent of hers (like color vision) has never been active, we'll grant her an excellent conscious understanding of that agent, and then we'll see what happens when we activate it for the first time.
But first, there's one more mental agent we need to introduce, one which serves a key purpose in Martha's evolutionary history: a simple agent that identifies learning.
We recall that Martha's species had evolved ever-better fluency with language, and along the way their minds developed a conscious/unconscious distinction, based on the functional difference (in the ancestral environment) between things that might be beneficial to communicate and things that weren't. Any new conscious node or connection has the potential to be communicated, but not all such elements are created equal; some of them (like a key bit of gossip, or a pattern suddenly grasped) have much larger effects on the full graph (and the conscious subgraph) than others.
Since the minds of Martha's species are similar in structure, it's likely that the new conscious ideas that rearrange Martha's graph might have similar effects on her friends' graphs- and therefore, Martha's species has evolved to give such connections high priority for communication.
One way to do this is to have a simple mental agent which tells Martha when she's learned something of significance. That is, it becomes active when the conscious subgraph rearranges on a large scale1. When this happens, it traces the cascade of changes backward until it finds the most basic new node or connection that started the revolution. When it finds the culprit, it forms the conscious idea "I just learned X", where X is the new node or connection; thus, when Martha is thinking or talking about related ideas, the important bit is more likely to occur to her.
In our toy example from yesterday's post, at the moment that the DREAD VILLAIN reveals himself as the LONE STALWART's father, a number of changes ripple through Martha's graph (once all the subconscious processes that transform patterns of sound into conscious concepts have gone through). The learning agent activates and seeks out the root cause; it finds the new connection that DV = AS, and the attached memory of the DREAD VILLAIN's mechanical bass voice revealing the secret. The learning agent creates a stronger memory of this idea and the moment it occurred, and strengthens the connection from these to the language centers. Then, when Martha retells the story, the nearby agents will trigger this particular memory (and Martha will include it unless she has other reasons not to). Everything functions as it ought.
So what happens when Martha has a fundamentally new experience?
Martha's Room
Let's put Martha in the situation of Mary's Room. She has all of the mental equipment for color vision, but has never had it activated by external experience. She has factual knowledge of the correspondence between particular objects and their colors. Her conscious graph has linked the names of cherry red and firetruck red and lime green to their proper RGB triplets, and knows which colors are between which. And she has factual knowledge of the structure of her subconscious visual-processing agents, and her mind in general- that is, her conscious graph contains a subgraph that is a good (though of course not infinitely detailed) model of the full graph, and this subgraph can model quite well what the full graph will do once color vision is activated.
At last, we introduce color vision for the first time. Vast subconscious agents form new connections to each other, rearranging Martha's mind in myriad ways; as in the third image in the previous post, the connections formed by subconscious agents add to the strength of the conscious connections, altering the conscious graph like a puppeteer moves a puppet. The learning agent activates and then starts looking for the culprit. It zeroes in on the epicenter of change in the conscious subgraph: Martha's representations of color knowledge.
And there it gets stuck. No new conscious connection formed between, say, "apple" and "red", and the learning agent is only looking for information in the conscious subgraph. The learning agent goes through a dozen false starts looking for something that isn't there. Martha articulates the thought of having learned something, but has nothing to say about what that new knowledge might be, except that it's something about the colors. This new knowledge is, somehow, ineffable to her.
However, this is no contradiction to her complete factual knowledge about the brain! Martha's mind has a conscious representation of what will happen to her mental graph once she sees colors, but this doesn't itself rearrange her subconscious visual processing agents, any more than drawing a new bridge on your map of California causes its construction in reality. (The potential for confusion of levels, it seems to me, gives the Mary's Room argument much of its persuasive force.)
Martha would not be surprised at this new rearrangement (since she could model in advance this snafu with her learning agent), but her mind would nonetheless have the reaction of learning something ineffable. No amount of prior factual knowledge would suffice to prevent the learning agent from malfunctioning in this way. Furthermore, this ineffability has the same boundaries as that we wanted to investigate: when Martha digests a new food, it doesn’t strongly affect agents that are part of the conscious subgraph, so the learning-agent isn't activated; when she counts sheep, anything she learns is a trivial development of her conscious knowledge of quantity2.
Since our model–which seems to be thoroughly reductionistic–gives rise to this reaction without paradox or mystery, the Mary's Room argument cannot be a logically valid argument against physical reductionism. But there's more to it than that...
Martha in the Mirror
Now that we've seen how the model gives rise to thoughts and reactions like those of human beings, why not carry the correspondence farther? There's perhaps no need for the Cartesian theater at all, if our ineffable qualia can arise from a similar interplay between vast subconscious agents and the conscious graph3. In the case of familiar qualia, we might look not at an agent which notices learning, but rather one which traces patterns of activation; as before, our conscious connections aren't enough to cause the pattern of activation by themselves, and another mental agent might well characterize this as ineffability in the same fashion.
But by focusing on the conscious thoughts of ineffability, are we neglecting the actual essence of the problem? It seems to me that once we remove the seeming paradox, the conclusion that different mental agents feel different to us is hardly a mysterious one. Your mileage may, perhaps, vary.
This is by no means a full or completed theory, but I find it promising enough as a start on the hard problem– and I think it might be helpful for those who find subjective experience a stumbling block on the way to reductionism. Thanks to several people on Less Wrong and elsewhere who've given me feedback on early versions, and thanks for all your thoughts now!
Addenda:
We should also consider the question of what we're actually doing when we think about the thought experiment, since we form our intuitions on Mary's Room well in advance of any experimental test. We're probably using our subconscious mental machinery to simulate what such a person might think and feel by empathetically feeling it ourselves, a process which is extremely useful on an evolutionary level for predicting other people's future actions, and which doesn't restrict itself to simulating only those aspects of psychology which we consciously understand4. (One might suspect that this ability to model other minds is the real origin of the recent arms race in brain size among hominids.)
Our subconscious model includes our conscious subgraph, the effect of a new sense experience, and the agents that recognize learning, and it rightly recognizes the feeling of ineffability that would result. But it's not designed to pass along a conscious understanding of the structure behind this feeling, so it's understandable that we naturally take it as a mysterious property of subjective experience rather than as a hiccup in our cognitive algorithms.
Finally, how much should we worry about Occam's Razor? Well, while the concepts I've introduced have taken some careful explanations, they're relatively basic entities which we could (if we wanted) program on a computer if we so chose. The phenomenon arises from the interaction of the following entities: a mental graph that rearranges upon new connections, the subgraph connected to language, and the simple learning agent. As far as cognitive science is concerned, we're being quite parsimonious.
Footnotes:
1. I'm positing that it focuses on the conscious subgraph, because there's no benefit to communicate events that only affect the agents that are useless to communicate about.
2. Come to think of it, I suppose there should be a qualia response to counting a new order of magnitude- "Ah! So that's what a crowd of 100,000 looks like." But intermediate numbers between known quantities should still be relatively qualia-free. It turns out to be really difficult to construct an example of mental processing that doesn't have a qualia aspect...
3. Our conscious/subconscious distinction may be a bit more complex than Martha's, but in the main it seems to correspond well to the boundary between what we evolved to communicate and what we did not.
4. Note that this is quite different from the conscious modeling of the human mind we posited for Mary and Martha. It's crucial, from an evolutionary perspective, that our subconscious models of human action aren't handicapped by the paltry state of our conscious notions of psychology. The model needs to actually get the right prediction to be successful.
"[I]magine that we have created computational intelligence in the form of an autonomous agent that perceives its environment and has the capacity to reflect rationally on what it perceives. What would such a system be like? Would it have any concept of consciousness, or any related notions?
"To see that it might, note that one the most natural design such a system would surely have some concept of self — for instance, it would have the ability to distinguish itself from the rest of the world, and from other entities resembling it. It also seems reasonable that such a system would be able to access its own cognitive contents much more directly than it could those of others. If it had the capacity to reflect, it would presumably have a certain direct awareness of its own thought contents, and could reason about that fact. Furthermore, such a system would most naturally have direct access to perceptual information, much as our own cognitive system does.
"When we asked the system what perception was like, what would it say? Would it say, "It's not like anything"? Might it say, "Well, I know there is a red tricycle over there, but I have no idea how I know it. The information just appeared in my database"? Perhaps, but it seems unlikely. A system designed this way would be curiously indirect. It seems much more likely that it would say, "I know there is a red tricycle because I see it there." When we ask it in turn how it knows that it is seeing the tricycle, the answer would very likely be something along the lines of "I just see it."
"It would be an odd system that replied, "I know I see it because sensors 78-84 are activated in such-and-such a way." As Hofstadter (1979) points out, there is no need to give a system such detailed access to its low-level parts. Even Winograd's program SHRDLU (1972) did not have knowledge about the code it was written in, despite the fact that it could perceive a virtual world, make inferences about the world, and even justify its knowledge to a limited degree. Such extra knowledge would seem to be quite unnecessary, and would only complicate the processes of awareness and inference.
"Instead, it seems likely that such a system would have the same kind of attitude toward its perceptual contents as we do toward ours, with its knowledge of them being directed and unmediated, at least as far as the system is concerned. When we ask how it knows that it sees the red tricycle, an efficiently designed system would say, "I just see it!" When we ask how it knows that the tricycle is red, it would say the same sort of thing that we do: "It just looks red." If such a system were reflective, it might start wondering about how it is that things look red, and about why it is that red just is a particular way, and blue another. From the system's point of view it is just a brute fact that red looks one way, and blue another. Of course from our vantage point we know that this is just because red throws the system into one state, and blue throws it into another; but from the machine's point of view this does not help.
"As it reflected, it might start to wonder about the very fact that it seems to have some access to what it is thinking, and that it has a sense of self. A reflective machine that was designed to have direct access to the contents of its perception and thought might very soon start wondering about the mysteries of consciousness (Hofstadter 1985a gives a rich discussion of this idea): "Why is it that heat feels this way?"; "Why am I me, and not someone else?"; "I know my processes are just electronic circuits, but how does this explain my experience of thought and perception?"
"Of course, the speculation I have engaged in here is not to be taken too seriously, but it helps to bring out the naturalness of the fact that we judge and claim that we are conscious, given a reasonable design. It would be a strange kind of cognitive system that had no idea what we were talking about when we asked what it was like to be it. The fact that we think and talk about consciousness may be a consequence of very natural features of our design, just as it is with these systems. And certainly, in the explanation of why these systems think and talk as they do, we will never need to invoke full-fledged consciousness. Perhaps these systems are really conscious and perhaps they are not, but the explanation works independently of this fact. Any explanation of how these systems function can be given solely in computational terms. In such a case it is obvious that there is no room for a ghost in the machine to play an explanatory role.
"All this is to say (expanding on a claim in Chapter 1) that consciousness is surprising, but claims about consciousness are not. Although consciousness is a feature of the world that we would not predict from the physical facts, the things we say about consciousness are a garden-variety cognitive phenomenon. Somebody who knew enough about cognitive structure would immediately be able to predict the likelihood of utterances such as "I feel conscious, in a way that no physical object could be," or even Descartes's "Cogito ergo sum." In principle, some reductive explanation in terms of internal processes should render claims about consciousness no more deeply surprising than any other aspect of behavior. [...]
"At this point a natural thought has probably occurred to many readers, especially those of a reductionist bent: If one has explained why we say we are conscious, and why we judge that we are conscious, haven't we explained all that there is to be explained? Why not simply give up on the quest for a theory of consciousness, declaring consciousness itself a chimera? Even better, why not declare one's theory of why we judge that we are conscious to be a theory of consciousness in its own right? It might well be suggested that a theory of our judgments is all the theory of consciousness that we need. [...]
"This is surely the single most powerful argument for a reductive or eliminative view of consciousness. But it is not enough. [...] Explaining our judgments about consciousness does not come close to removing the mysteries of consciousness. Why? Because consciousness is itself an explanandum. The existence of God was arguably hypothesized largely in order to explain all sorts of evident facts about the world, such as its orderliness and its apparent design. When it turns out that an alternative hypothesis can explain the evidence just as well, then there is no need for the hypothesis of God. There is no separate phenomenon God that we can point to and say: that needs explaining. At best, there is indirect evidence. [...]
"But consciousness is not an explanatory construct, postulated to help explain behavior or events in the world. Rather, it is a brute explanandum, a phenomenon in its own right that is in need of explanation. It therefore does not matter if it turns out that consciousness is not required to do any work in explaining other phenomena. Our evidence for consciousness never lay with these other phenomena in the first place. Even if our judgments about consciousness are reductively explained, all this shows is that our judgments can be explained reductively. The mind-body problem is not that of explaining our judgments about consciousness. If it were, it would be a relatively trivial problem. Rather, the mind-body problem is that of explaining consciousness itself. If the judgments can be explained without explaining consciousness, then that is interesting and perhaps surprising, but it does not remove the mind-body problem.
"To take the line that explaining our judgments about consciousness is enough [...] is most naturally understood as an eliminativist position about consciousness [...]. As such it suffers from all the problems that eliminativism naturally faces. In particular, it denies the evidence of our own experience. This is the sort of thing that can only be done by a philosopher — or by someone else tying themselves in intellectual knots. Our experiences of red do not go away upon making such a denial. It is still like something to be us, and that is still something that needs explanation. [...]
"There is a certain intellectual appeal to the position that explaining phenomenal judgments is enough. It has the feel of a bold stroke that cleanly dissolves all the problems, leaving our confusion lying on the ground in front of us exposed for all to see. Yet it is the kind of "solution" that is satisfying only for about half a minute. When we stop to reflect, we realize that all we have done is to explain certain aspects of behavior. We have explained why we talk in certain ways, and why we are disposed to do so, but we have not remotely come to grips with the central problem, namely conscious experience itself. When thirty seconds are up, we find ourselves looking at a red rose, inhaling its fragrance, and wondering: "Why do I experience it like this?" And we realize that this explanation has nothing to say about the matter. [...]
"This line of argument is perhaps the most interesting that a reductionist or eliminativist can take — if I were a reductionist, I would be this sort of reductionist — but at the end of the day it suffers from the problem that all such positions face: it does not explain what needs to be explained. Tempting as this position is, it ends up failing to take the problem seriously. The puzzle of consciousness cannot be removed by such simple means."
—David Chalmers, The Conscious Mind: In Search of a Fundamental Theory (1996)
A shorter way of getting to the same conclusion.
If Orthonormal's argument summarises to
Then the objection would be