After reading the first post in this sequence, I had the idea that a quale is like a handle to a kernel object in programming. Subconscious brain corresponds to the OS kernel, and conscious brain corresponds to user-space. When you see red, you get a handle to a "redness" object, which you can perform certain queries and operations on, such as "does this make me feel hot or cold", or "how similar is this color to this other color" but you can't directly access the underlying data structure. Nor can the conscious brain cause the redness object to be serialized into a description that can be deserialized in another brain to recreate the object. Nor can Mary instantiate a redness object in her brain by studying neuroscience.

Orthonormal, does this model also capture what you're saying, or does the "graph" model offer different insights?

And there it gets stuck. No new conscious connection formed between, say, "apple" and "red", and the learning agent is only looking for information in the conscious subgraph. The learning agent goes through a dozen false starts looking for something that isn't there. Martha articulates the thought of having learned something, but has nothing to say about what that new knowledge might be, except that it's something about the colors. This new knowledge is, somehow, ineffable to her.

Why do you state that Martha "has nothing to say" about her new knowledge? Here's an alternate possibility: the color pathways in her brain will light up, and her mental agents will start processing color information. As part of this processing, new nodes will appear in the 'conscious' semantic network corresponding to the principal components in said information, i.e. the main colors (including red); these nodes will subsequently collapse with the existing nodes for color words. As in the OP, Martha has the feeling of having learned something, but she can confidently state that she has "learned what red looks like".

This is in fact what happens when "extra senses" are added to the body by exploiting the adult brain's plasticity, such as with a magnetic belt (giving an "orientation sense") or by inserting small magnets under the skin.

Excellent article. Walking through the scenario with these toy graph-based conscious entities substituted for the humans is a good first step that is likely to help with other conundrums like this.

So if this is what qualia are, any simulated human-like being that has both conscious and nontrivial unconscious processing has qualia, right?

So in short, "surprise" is not necessarily an indication of lack of "knowledge".

Sounds cool any ideas on how we can test it experimentally? The theory fits with what I have already obsevered about quailia and does technically allow me to make predictions in advance, but all of the ones I can both think of and observe are ones I would have already been able to make based off of prior experience with qualia. There's a good chance thats just because I have a head ache and am not thinking to clearly, but I would still be interested in hearing if you have any predictions that you would care to make that aren't already clear from peoples intuitive understanding of qualia/other formal theories. I can thing of some new predictions regarding what I would expect to see if I directly obsevered the functioning of the human brain more closely than I am actually capable of.

I haven't read the post yet, but the title is awesome.

orthonormal, is the following an accurate summary of your approach to the Mary's Room paradox?

Certain states of Mary's conscious mind cannot be brought about by any action of her conscious mind. In particular, no sequence of words can put her conscious mind into the state of experiencing red. But effable experiences are just those states of consciousness that can be induced by sequences of words. Therefore, Mary will find the experience of seeing red to be ineffable.

One might suspect that this ability to model other minds is the real origin of the recent arms race in brain size among hominids.

Surely chimps do that too. The primary cognitive attribute that we have but chimps mostly don't is cultural inheritance. The picture there is one of apes with infected brains, whose brain cases swelled up to provide more room for their mutualist breathren. This is quite a different picture.

I've just had some new insights, which may necessitate editing the post or even writing an additional one; I've started to think that the "Martha in the Mirror" section above is much weaker than the rest of my sequence, and I might have the idea now of how to fix that.

In the meantime, I've not received as much feedback (negative or positive) as I'd hoped, and I'm wondering why. If you could help me out with the following poll, I'd appreciate it...

(As usual, please only vote an option up, not down, so I can get an accurate tally. The last option is a karma equalizer for you to vote down so that I'm not grabbing free upvotes.)

If the learning agent does not find any new knowledge why does it make Martha report having learned something new? Why not make her feel as if nothing changed?

"[I]magine that we have created computational intelligence in the form of an autonomous agent that perceives its environment and has the capacity to reflect rationally on what it perceives. What would such a system be like? Would it have any concept of consciousness, or any related notions?

"To see that it might, note that one the most natural design such a system would surely have some concept of self — for instance, it would have the ability to distinguish itself from the rest of the world, and from other entities resembling it. It also seems reasonable that such a system would be able to access its own cognitive contents much more directly than it could those of others. If it had the capacity to reflect, it would presumably have a certain direct awareness of its own thought contents, and could reason about that fact. Furthermore, such a system would most naturally have direct access to perceptual information, much as our own cognitive system does.

"When we asked the system what perception was like, what would it say? Would it say, "It's not like anything"? Might it say, "Well, I know there is a red tricycle over there, but I have no idea how I know it. The information just appeared in my database"? Perhaps, but it seems unlikely. A system designed this way would be curiously indirect. It seems much more likely that it would say, "I know there is a red tricycle because I see it there." When we ask it in turn how it knows that it is seeing the tricycle, the answer would very likely be something along the lines of "I just see it."

"It would be an odd system that replied, "I know I see it because sensors 78-84 are activated in such-and-such a way." As Hofstadter (1979) points out, there is no need to give a system such detailed access to its low-level parts. Even Winograd's program SHRDLU (1972) did not have knowledge about the code it was written in, despite the fact that it could perceive a virtual world, make inferences about the world, and even justify its knowledge to a limited degree. Such extra knowledge would seem to be quite unnecessary, and would only complicate the processes of awareness and inference.

"Instead, it seems likely that such a system would have the same kind of attitude toward its perceptual contents as we do toward ours, with its knowledge of them being directed and unmediated, at least as far as the system is concerned. When we ask how it knows that it sees the red tricycle, an efficiently designed system would say, "I just see it!" When we ask how it knows that the tricycle is red, it would say the same sort of thing that we do: "It just looks red." If such a system were reflective, it might start wondering about how it is that things look red, and about why it is that red just is a particular way, and blue another. From the system's point of view it is just a brute fact that red looks one way, and blue another. Of course from our vantage point we know that this is just because red throws the system into one state, and blue throws it into another; but from the machine's point of view this does not help.

"As it reflected, it might start to wonder about the very fact that it seems to have some access to what it is thinking, and that it has a sense of self. A reflective machine that was designed to have direct access to the contents of its perception and thought might very soon start wondering about the mysteries of consciousness (Hofstadter 1985a gives a rich discussion of this idea): "Why is it that heat feels this way?"; "Why am I me, and not someone else?"; "I know my processes are just electronic circuits, but how does this explain my experience of thought and perception?"

"Of course, the speculation I have engaged in here is not to be taken too seriously, but it helps to bring out the naturalness of the fact that we judge and claim that we are conscious, given a reasonable design. It would be a strange kind of cognitive system that had no idea what we were talking about when we asked what it was like to be it. The fact that we think and talk about consciousness may be a consequence of very natural features of our design, just as it is with these systems. And certainly, in the explanation of why these systems think and talk as they do, we will never need to invoke full-fledged consciousness. Perhaps these systems are really conscious and perhaps they are not, but the explanation works independently of this fact. Any explanation of how these systems function can be given solely in computational terms. In such a case it is obvious that there is no room for a ghost in the machine to play an explanatory role.

"All this is to say (expanding on a claim in Chapter 1) that consciousness is surprising, but claims about consciousness are not. Although consciousness is a feature of the world that we would not predict from the physical facts, the things we say about consciousness are a garden-variety cognitive phenomenon. Somebody who knew enough about cognitive structure would immediately be able to predict the likelihood of utterances such as "I feel conscious, in a way that no physical object could be," or even Descartes's "Cogito ergo sum." In principle, some reductive explanation in terms of internal processes should render claims about consciousness no more deeply surprising than any other aspect of behavior. [...]

"At this point a natural thought has probably occurred to many readers, especially those of a reductionist bent: If one has explained why we say we are conscious, and why we judge that we are conscious, haven't we explained all that there is to be explained? Why not simply give up on the quest for a theory of consciousness, declaring consciousness itself a chimera? Even better, why not declare one's theory of why we judge that we are conscious to be a theory of consciousness in its own right? It might well be suggested that a theory of our judgments is all the theory of consciousness that we need. [...]

"This is surely the single most powerful argument for a reductive or eliminative view of consciousness. But it is not enough. [...] Explaining our judgments about consciousness does not come close to removing the mysteries of consciousness. Why? Because consciousness is itself an explanandum. The existence of God was arguably hypothesized largely in order to explain all sorts of evident facts about the world, such as its orderliness and its apparent design. When it turns out that an alternative hypothesis can explain the evidence just as well, then there is no need for the hypothesis of God. There is no separate phenomenon God that we can point to and say: that needs explaining. At best, there is indirect evidence. [...]

"But consciousness is not an explanatory construct, postulated to help explain behavior or events in the world. Rather, it is a brute explanandum, a phenomenon in its own right that is in need of explanation. It therefore does not matter if it turns out that consciousness is not required to do any work in explaining other phenomena. Our evidence for consciousness never lay with these other phenomena in the first place. Even if our judgments about consciousness are reductively explained, all this shows is that our judgments can be explained reductively. The mind-body problem is not that of explaining our judgments about consciousness. If it were, it would be a relatively trivial problem. Rather, the mind-body problem is that of explaining consciousness itself. If the judgments can be explained without explaining consciousness, then that is interesting and perhaps surprising, but it does not remove the mind-body problem.

"To take the line that explaining our judgments about consciousness is enough [...] is most naturally understood as an eliminativist position about consciousness [...]. As such it suffers from all the problems that eliminativism naturally faces. In particular, it denies the evidence of our own experience. This is the sort of thing that can only be done by a philosopher — or by someone else tying themselves in intellectual knots. Our experiences of red do not go away upon making such a denial. It is still like something to be us, and that is still something that needs explanation. [...]

"There is a certain intellectual appeal to the position that explaining phenomenal judgments is enough. It has the feel of a bold stroke that cleanly dissolves all the problems, leaving our confusion lying on the ground in front of us exposed for all to see. Yet it is the kind of "solution" that is satisfying only for about half a minute. When we stop to reflect, we realize that all we have done is to explain certain aspects of behavior. We have explained why we talk in certain ways, and why we are disposed to do so, but we have not remotely come to grips with the central problem, namely conscious experience itself. When thirty seconds are up, we find ourselves looking at a red rose, inhaling its fragrance, and wondering: "Why do I experience it like this?" And we realize that this explanation has nothing to say about the matter. [...]

"This line of argument is perhaps the most interesting that a reductionist or eliminativist can take — if I were a reductionist, I would be this sort of reductionist — but at the end of the day it suffers from the problem that all such positions face: it does not explain what needs to be explained. Tempting as this position is, it ends up failing to take the problem seriously. The puzzle of consciousness cannot be removed by such simple means."

—David Chalmers, The Conscious Mind: In Search of a Fundamental Theory (1996)

The color red has many associations to the subconscious. These associations lie dormant and unused until the first time she actually experiences the color red. No amount of study could enable her to understand every nuance of these associations to the color red. I think that she really would learn something upon first seeing the color red.

Those things would probably not be particularly important because if they were someone would have taken the time to write them down at some point and Martha would have read about them. Things like "The color red is associated with aggression" are easy enough to learn but they are only qualitative, not quantitative. Until she actually experiences the color she will have no idea exactly how strong those associations are.

Perceiving a color has many very subtle effects on the mind that are not easy to identify even while you are experiencing them. When Martha first perceives the color red she experiences these connections for the first time. I think this would suffice to explain that Martha feels like she has learned something, because she actually has learned something, it's just too subtle to point out exactly what it is.

That said, I think you are right in principle in that different parts of the mind/ mental agents/ memes can work against each other by mistake or be harmful to the mind as a whole because they don't realize that they don't apply to a given situation. I just think that this particular scenario has a different explanation.