LESSWRONG
LW

Comment Permalink
Rob Bensinger-10

"[I]magine that we have created computational intelligence in the form of an autonomous agent that perceives its environment and has the capacity to reflect rationally on what it perceives. What would such a system be like? Would it have any concept of consciousness, or any related notions?

"To see that it might, note that one the most natural design such a system would surely have some concept of self — for instance, it would have the ability to distinguish itself from the rest of the world, and from other entities resembling it. It also seems reasonable that such a system would be able to access its own cognitive contents much more directly than it could those of others. If it had the capacity to reflect, it would presumably have a certain direct awareness of its own thought contents, and could reason about that fact. Furthermore, such a system would most naturally have direct access to perceptual information, much as our own cognitive system does.

"When we asked the system what perception was like, what would it say? Would it say, "It's not like anything"? Might it say, "Well, I know there is a red tricycle over there, but I have no idea how I know it. The information just appeared in my database"? Perhaps, but it seems unlikely. A system designed this way would be curiously indirect. It seems much more likely that it would say, "I know there is a red tricycle because I see it there." When we ask it in turn how it knows that it is seeing the tricycle, the answer would very likely be something along the lines of "I just see it."

"It would be an odd system that replied, "I know I see it because sensors 78-84 are activated in such-and-such a way." As Hofstadter (1979) points out, there is no need to give a system such detailed access to its low-level parts. Even Winograd's program SHRDLU (1972) did not have knowledge about the code it was written in, despite the fact that it could perceive a virtual world, make inferences about the world, and even justify its knowledge to a limited degree. Such extra knowledge would seem to be quite unnecessary, and would only complicate the processes of awareness and inference.

"Instead, it seems likely that such a system would have the same kind of attitude toward its perceptual contents as we do toward ours, with its knowledge of them being directed and unmediated, at least as far as the system is concerned. When we ask how it knows that it sees the red tricycle, an efficiently designed system would say, "I just see it!" When we ask how it knows that the tricycle is red, it would say the same sort of thing that we do: "It just looks red." If such a system were reflective, it might start wondering about how it is that things look red, and about why it is that red just is a particular way, and blue another. From the system's point of view it is just a brute fact that red looks one way, and blue another. Of course from our vantage point we know that this is just because red throws the system into one state, and blue throws it into another; but from the machine's point of view this does not help.

"As it reflected, it might start to wonder about the very fact that it seems to have some access to what it is thinking, and that it has a sense of self. A reflective machine that was designed to have direct access to the contents of its perception and thought might very soon start wondering about the mysteries of consciousness (Hofstadter 1985a gives a rich discussion of this idea): "Why is it that heat feels this way?"; "Why am I me, and not someone else?"; "I know my processes are just electronic circuits, but how does this explain my experience of thought and perception?"

"Of course, the speculation I have engaged in here is not to be taken too seriously, but it helps to bring out the naturalness of the fact that we judge and claim that we are conscious, given a reasonable design. It would be a strange kind of cognitive system that had no idea what we were talking about when we asked what it was like to be it. The fact that we think and talk about consciousness may be a consequence of very natural features of our design, just as it is with these systems. And certainly, in the explanation of why these systems think and talk as they do, we will never need to invoke full-fledged consciousness. Perhaps these systems are really conscious and perhaps they are not, but the explanation works independently of this fact. Any explanation of how these systems function can be given solely in computational terms. In such a case it is obvious that there is no room for a ghost in the machine to play an explanatory role.

"All this is to say (expanding on a claim in Chapter 1) that consciousness is surprising, but claims about consciousness are not. Although consciousness is a feature of the world that we would not predict from the physical facts, the things we say about consciousness are a garden-variety cognitive phenomenon. Somebody who knew enough about cognitive structure would immediately be able to predict the likelihood of utterances such as "I feel conscious, in a way that no physical object could be," or even Descartes's "Cogito ergo sum." In principle, some reductive explanation in terms of internal processes should render claims about consciousness no more deeply surprising than any other aspect of behavior. [...]

"At this point a natural thought has probably occurred to many readers, especially those of a reductionist bent: If one has explained why we say we are conscious, and why we judge that we are conscious, haven't we explained all that there is to be explained? Why not simply give up on the quest for a theory of consciousness, declaring consciousness itself a chimera? Even better, why not declare one's theory of why we judge that we are conscious to be a theory of consciousness in its own right? It might well be suggested that a theory of our judgments is all the theory of consciousness that we need. [...]

"This is surely the single most powerful argument for a reductive or eliminative view of consciousness. But it is not enough. [...] Explaining our judgments about consciousness does not come close to removing the mysteries of consciousness. Why? Because consciousness is itself an explanandum. The existence of God was arguably hypothesized largely in order to explain all sorts of evident facts about the world, such as its orderliness and its apparent design. When it turns out that an alternative hypothesis can explain the evidence just as well, then there is no need for the hypothesis of God. There is no separate phenomenon God that we can point to and say: that needs explaining. At best, there is indirect evidence. [...]

"But consciousness is not an explanatory construct, postulated to help explain behavior or events in the world. Rather, it is a brute explanandum, a phenomenon in its own right that is in need of explanation. It therefore does not matter if it turns out that consciousness is not required to do any work in explaining other phenomena. Our evidence for consciousness never lay with these other phenomena in the first place. Even if our judgments about consciousness are reductively explained, all this shows is that our judgments can be explained reductively. The mind-body problem is not that of explaining our judgments about consciousness. If it were, it would be a relatively trivial problem. Rather, the mind-body problem is that of explaining consciousness itself. If the judgments can be explained without explaining consciousness, then that is interesting and perhaps surprising, but it does not remove the mind-body problem.

"To take the line that explaining our judgments about consciousness is enough [...] is most naturally understood as an eliminativist position about consciousness [...]. As such it suffers from all the problems that eliminativism naturally faces. In particular, it denies the evidence of our own experience. This is the sort of thing that can only be done by a philosopher — or by someone else tying themselves in intellectual knots. Our experiences of red do not go away upon making such a denial. It is still like something to be us, and that is still something that needs explanation. [...]

"There is a certain intellectual appeal to the position that explaining phenomenal judgments is enough. It has the feel of a bold stroke that cleanly dissolves all the problems, leaving our confusion lying on the ground in front of us exposed for all to see. Yet it is the kind of "solution" that is satisfying only for about half a minute. When we stop to reflect, we realize that all we have done is to explain certain aspects of behavior. We have explained why we talk in certain ways, and why we are disposed to do so, but we have not remotely come to grips with the central problem, namely conscious experience itself. When thirty seconds are up, we find ourselves looking at a red rose, inhaling its fragrance, and wondering: "Why do I experience it like this?" And we realize that this explanation has nothing to say about the matter. [...]

"This line of argument is perhaps the most interesting that a reductionist or eliminativist can take — if I were a reductionist, I would be this sort of reductionist — but at the end of the day it suffers from the problem that all such positions face: it does not explain what needs to be explained. Tempting as this position is, it ends up failing to take the problem seriously. The puzzle of consciousness cannot be removed by such simple means."

—David Chalmers, The Conscious Mind: In Search of a Fundamental Theory (1996)

Reply
TheAncientGeek-10

A shorter way of getting to the same conclusion.

If Orthonormal's argument summarises to

  1. Qualia have the property of ineffability, and no other. They are just cognitive blind spots.
  2. Ineffability can be explained reductionistically.
  3. Therefore, qualia can be explained reductionasitically.

Then the objection would be

  1. Qualia are crucial to morally relevance -- it is not OK to torture sims that actually feel pain.
  2. Ineffability, the possession of cognitive blind spots is not morally relevant. ("How dare you torture that sim, don't you realise it is unable
... (read more)
Reply
See in context
by orthonormal
7 min read
64

62

Previously: Seeing Red: Dissolving Mary's Room and Qualia, A Study of Scarlet: The Conscious Mental Graph

When we left off, we'd introduced a hypothetical organism called Martha whose actions are directed by a mobile graph of simple mental agents. The tip of the iceberg, consisting of the agents that are connected to Martha's language centers, we called the conscious subgraph. Now we're going to place Martha into a situation like Mary's Room: we'll say that a large unconscious agent of hers (like color vision) has never been active, we'll grant her an excellent conscious understanding of that agent, and then we'll see what happens when we activate it for the first time.

But first, there's one more mental agent we need to introduce, one which serves a key purpose in Martha's evolutionary history: a simple agent that identifies learning.

We recall that Martha's species had evolved ever-better fluency with language, and along the way their minds developed a conscious/unconscious distinction, based on the functional difference (in the ancestral environment) between things that might be beneficial to communicate and things that weren't. Any new conscious node or connection has the potential to be communicated, but not all such elements are created equal; some of them (like a key bit of gossip, or a pattern suddenly grasped) have much larger effects on the full graph (and the conscious subgraph) than others.

Since the minds of Martha's species are similar in structure, it's likely that the new conscious ideas that rearrange Martha's graph might have similar effects on her friends' graphs- and therefore, Martha's species has evolved to give such connections high priority for communication.

One way to do this is to have a simple mental agent which tells Martha when she's learned something of significance. That is, it becomes active when the conscious subgraph rearranges on a large scale1. When this happens, it traces the cascade of changes backward until it finds the most basic new node or connection that started the revolution. When it finds the culprit, it forms the conscious idea "I just learned X", where X is the new node or connection; thus, when Martha is thinking or talking about related ideas, the important bit is more likely to occur to her.

In our toy example from yesterday's post, at the moment that the DREAD VILLAIN reveals himself as the LONE STALWART's father, a number of changes ripple through Martha's graph (once all the subconscious processes that transform patterns of sound into conscious concepts have gone through). The learning agent activates and seeks out the root cause; it finds the new connection that DV = AS, and the attached memory of the DREAD VILLAIN's mechanical bass voice revealing the secret. The learning agent creates a stronger memory of this idea and the moment it occurred, and strengthens the connection from these to the language centers. Then, when Martha retells the story, the nearby agents will trigger this particular memory (and Martha will include it unless she has other reasons not to). Everything functions as it ought.

So what happens when Martha has a fundamentally new experience?

Martha's Room

Let's put Martha in the situation of Mary's Room. She has all of the mental equipment for color vision, but has never had it activated by external experience. She has factual knowledge of the correspondence between particular objects and their colors. Her conscious graph has linked the names of cherry red and firetruck red and lime green to their proper RGB triplets, and knows which colors are between which. And she has factual knowledge of the structure of her subconscious visual-processing agents, and her mind in general- that is, her conscious graph contains a subgraph that is a good (though of course not infinitely detailed) model of the full graph, and this subgraph can model quite well what the full graph will do once color vision is activated.

At last, we introduce color vision for the first time. Vast subconscious agents form new connections to each other, rearranging Martha's mind in myriad ways; as in the third image in the previous post, the connections formed by subconscious agents add to the strength of the conscious connections, altering the conscious graph like a puppeteer moves a puppet. The learning agent activates and then starts looking for the culprit. It zeroes in on the epicenter of change in the conscious subgraph: Martha's representations of color knowledge.

And there it gets stuck. No new conscious connection formed between, say, "apple" and "red", and the learning agent is only looking for information in the conscious subgraph. The learning agent goes through a dozen false starts looking for something that isn't there. Martha articulates the thought of having learned something, but has nothing to say about what that new knowledge might be, except that it's something about the colors. This new knowledge is, somehow, ineffable to her.

However, this is no contradiction to her complete factual knowledge about the brain! Martha's mind has a conscious representation of what will happen to her mental graph once she sees colors, but this doesn't itself rearrange her subconscious visual processing agents, any more than drawing a new bridge on your map of California causes its construction in reality. (The potential for confusion of levels, it seems to me, gives the Mary's Room argument much of its persuasive force.)

Martha would not be surprised at this new rearrangement (since she could model in advance this snafu with her learning agent), but her mind would nonetheless have the reaction of learning something ineffable. No amount of prior factual knowledge would suffice to prevent the learning agent from malfunctioning in this way. Furthermore, this ineffability has the same boundaries as that we wanted to investigate: when Martha digests a new food, it doesn’t strongly affect agents that are part of the conscious subgraph, so the learning-agent isn't activated; when she counts sheep, anything she learns is a trivial development of her conscious knowledge of quantity2.

Since our model–which seems to be thoroughly reductionistic–gives rise to this reaction without paradox or mystery, the Mary's Room argument cannot be a logically valid argument against physical reductionism. But there's more to it than that...

Martha in the Mirror

Now that we've seen how the model gives rise to thoughts and reactions like those of human beings, why not carry the correspondence farther? There's perhaps no need for the Cartesian theater at all, if our ineffable qualia can arise from a similar interplay between vast subconscious agents and the conscious graph3. In the case of familiar qualia, we might look not at an agent which notices learning, but rather one which traces patterns of activation; as before, our conscious connections aren't enough to cause the pattern of activation by themselves, and another mental agent might well characterize this as ineffability in the same fashion.

But by focusing on the conscious thoughts of ineffability, are we neglecting the actual essence of the problem? It seems to me that once we remove the seeming paradox, the conclusion that different mental agents feel different to us is hardly a mysterious one. Your mileage may, perhaps, vary.

This is by no means a full or completed theory, but I find it promising enough as a start on the hard problem– and I think it might be helpful for those who find subjective experience a stumbling block on the way to reductionism. Thanks to several people on Less Wrong and elsewhere who've given me feedback on early versions, and thanks for all your thoughts now!

Addenda:

We should also consider the question of what we're actually doing when we think about the thought experiment, since we form our intuitions on Mary's Room well in advance of any experimental test. We're probably using our subconscious mental machinery to simulate what such a person might think and feel by empathetically feeling it ourselves, a process which is extremely useful on an evolutionary level for predicting other people's future actions, and which doesn't restrict itself to simulating only those aspects of psychology which we consciously understand4. (One might suspect that this ability to model other minds is the real origin of the recent arms race in brain size among hominids.)

Our subconscious model includes our conscious subgraph, the effect of a new sense experience, and the agents that recognize learning, and it rightly recognizes the feeling of ineffability that would result. But it's not designed to pass along a conscious understanding of the structure behind this feeling, so it's understandable that we naturally take it as a mysterious property of subjective experience rather than as a hiccup in our cognitive algorithms.

Finally, how much should we worry about Occam's Razor? Well, while the concepts I've introduced have taken some careful explanations, they're relatively basic entities which we could (if we wanted) program on a computer if we so chose. The phenomenon arises from the interaction of the following entities: a mental graph that rearranges upon new connections, the subgraph connected to language, and the simple learning agent. As far as cognitive science is concerned, we're being quite parsimonious.

Footnotes:

1. I'm positing that it focuses on the conscious subgraph, because there's no benefit to communicate events that only affect the agents that are useless to communicate about.

2. Come to think of it, I suppose there should be a qualia response to counting a new order of magnitude- "Ah! So that's what a crowd of 100,000 looks like." But intermediate numbers between known quantities should still be relatively qualia-free. It turns out to be really difficult to construct an example of mental processing that doesn't have a qualia aspect...

3. Our conscious/subconscious distinction may be a bit more complex than Martha's, but in the main it seems to correspond well to the boundary between what we evolved to communicate and what we did not.

4. Note that this is quite different from the conscious modeling of the human mind we posited for Mary and Martha. It's crucial, from an evolutionary perspective, that our subconscious models of human action aren't handicapped by the paltry state of our conscious notions of psychology. The model needs to actually get the right prediction to be successful.

ConsciousnessQualia
Frontpage

62

Mentioned in
Nature: Red, in Truth and Qualia
New Comment
64 comments, sorted by
top scoring
Click to highlight new comments since:
[-]Wei Dai150

After reading the first post in this sequence, I had the idea that a quale is like a handle to a kernel object in programming. Subconscious brain corresponds to the OS kernel, and conscious brain corresponds to user-space. When you see red, you get a handle to a "redness" object, which you can perform certain queries and operations on, such as "does this make me feel hot or cold", or "how similar is this color to this other color" but you can't directly access the underlying data structure. Nor can the conscious brain cause the redness object to be serialized into a description that can be deserialized in another brain to recreate the object. Nor can Mary instantiate a redness object in her brain by studying neuroscience.

Orthonormal, does this model also capture what you're saying, or does the "graph" model offer different insights?

Reply
[-]Vladimir_Nesov50

This is basically how Drescher explains qualia in Good and Real (section 2.5.2). He makes a reference to gensyms, in this context tags that can only be tested for equality and make no other semantics available.

Reply
[-]Wei Dai60

I had read that, but it didn't "click" with me, probably because I had never programed in Lisp before. I'm guessing the OS handle analogy would make more intuitive sense to the typical programmer.

BTW, in this line of thinking, there's nothing really special or mysterious about consciousness. It's just a mind or a part of a mind with verbal abilities. But that seems to rule out the approach of defining pain (as a moral disvalue) as negative reinforcement experienced by a conscious mind. If consciousness is just possession of verbal abilities, there seems no reason to assign special moral status to it. But then what is pain, and what is morally bad about it?

Reply
[-]orthonormal80

A related and horrible thought: has anyone thought to ask the right lobe of a split-brain patient whether it's in pain? Or, more horribly: has anyone tried to suppress the corpus callosum for a moment (there's a specialized helmet which targets electromagnetic waves to do this for various areas) and ask the right lobe of an ordinary person how they feel?

Reply
[-]Richard_Kennaway50

BTW, in this line of thinking, there's nothing really special or mysterious about consciousness. It's just a mind or a part of a mind with verbal abilities.

Huh? I am conscious of a lot of things that don't involve words. In fact, words are a very small part of my awareness, except when I'm actually using them, e.g. composing a comment like this one. They're a major way of communicating one's state of mind to someone else, but to identify consciousness with that facility would be like mistaking a computer screen for the computer.

I suppose that might change if one were to practice one of those meditation exercises in which one deliberately labels every moment of experience; but I don't know why one would want to do that.

Reply
[-]Wei Dai00

I'm not sure that identifying consciousness with access to verbal ability is the right way to go, but it is the approach taken "in this line of thinking", and I was trying to explore its implications. Quoting from the second post in orthonormal's sequence:

So within Martha's graph, there's a relatively small subgraph that's hooked up to the language areas; we'll call this her conscious subgraph.

I notice that you didn't object when orthonormal wrote that. Do you think what I wrote is substantially different, or ....? (ETA: we can discuss your disagreement with this approach too, but first I'd like to know whether I've misunderstood orthonormal somehow.)

Reply
[-]Richard_Kennaway00

I notice that you didn't object when orthonormal wrote that.

I don't necessarily read everything on LW. I didn't pay all that much attention to the original post, because a wall of text on yet another thought experiment designed to elucidate the nature of qualia wasn't something I wanted to read. But I noticed your comment, and my response, had I made one, to orthonormal's "we'll call this her conscious subgraph" would have been on the same lines.

I'm not sure that identifying consciousness with access to verbal ability is the right way to go, but it is the approach taken "in this line of thinking", and I was trying to explore its implications.

I think that this line of thinking leads nowhere, as every other line of thinking on the subject has done. Maybe I should write a top-level post on this, but it seems to me that no-one, on LessWrong or anywhere else that I've read, has explicitly faced up to the basic problem of consciousness/subjective experience/qualia/whatever you want to call it. You have the unstoppable force of materialism, the fundamental insight that "it's all atoms!", which has successfully solved or dissolved so many questions about the world, and then you have the immovable rock of conscious experience. The two are starkly incompatible. The question is too big to Ignore, Worshipping the mystery isn't an option (and is just a way of Ignoring while pretending not to), but no-one has any idea of what an Explanation could even look like. Every purported explanation of consciousness, on closer examination, turns out to be an explanation of something else, such as how an unconscious system might come to make assertions about consciousness.

I don't have a solution either. Explain, Worship, or Ignore? I can't hit any of those buttons.

Reply
[-]johnlawrenceaspden00

I'd seriously like to read such a post, were you to get round to writing it. Something very odd is going on, it seems to me, but I can't even express the problem.

Reply
[-]Will_Sawin00

I think it's more accurate to say that the part of our mind that is conscious is the part that uses words, since when you want to tell if someone knows something consciously or unconsciously, you ask them about it.

Reply
[-]Richard_Kennaway00

That's the same fallacy as I was trying to get at. I am conscious of a thing, or not, whether or not I tell anyone else, and it is easy to think of situations in which I can tell whether someone else is conscious of something without their saying anything. Words can tell you a lot, but they're telling you about something which is not those words, and is not the mechanism of producing those words.

Reply
[-]EniScien10

It was too difficult, but i want to talk it, so i use Google Translate: If consciousness is simply the ability to talk about the state of one's mind, accept this information from another, and hear one's own virtual story, then it is not clear why it should really make pain more meaningful. The evolutionary reason is that consciousness/talk (and if it's just introspection, I know what it feels like to turn on consciousness, sometimes it happens between sleep and reality, more precisely, this feeling of anti-disorientation, it feels like "pum" - turned on, and now you see the consequences of the previous lack of self-reflection) this is a distinguishing feature of people, people have similar genes and therefore those people who seek to protect other people who, unlike animals, are more likely to have these genes, were selected. However, if we do not proceed from evolutionary values, then no matter how alien it may be, consciousness really does not have much value, it’s just that conscious intellects are interpreted by us as living, not pieces of iron, so we may turn on a sense of empathy for them, but for tormented by pain and unable to do anything else, the computer and its protection are activated in us by a feeling of strangeness, strangeness and wrongness. However, outside of this, consciousness still makes beings more valuable, because it is a lens that sees its defects, it has reflection, it is a mirror that reflects itself, so the pain of conscious beings will be about twice as much, although it all depends on the number of levels of reflection and detail reflection of previous levels.

Reply
[-]Vladimir_Nesov00

I don't understand the transition of this conversation to consciousness, or what motivates the particular "verbal abilities" reference you've made. What do you mean by "in this line of thinking"? (I didn't really follow the posts.)

(With consciousness, I too prefer Drescher's unpacking, where the word refers to observation of the process of thinking (which is part of the thinking and can in turn be observed).)

Reply
[-]Wei Dai00

What do you mean by "in this line of thinking"? (I didn't really follow the posts.)

I was referring to orthonormal's unpacking of consciousness. (If people comment under posts that they have not read, could they please say so right away, instead of waiting until confusion arises? Sorry, but I'm a bit frustrated that both you and my other correspondent in this thread didn't read the posts and failed to say so earlier.)

(With consciousness, I too prefer Drescher's unpacking, where the word refers to observation of the process of thinking (which is part of the thinking and can in turn be observed).)

Unfortunately, that also does not seem to help with understanding pain as a moral disvalue, since it's unclear why we should assign special moral status to minds that can observe their own thinking.

Reply
[-]Vladimir_Nesov00

If people comment under posts that they have not read, could they please say so right away, instead of waiting until confusion arises?

My previous comment didn't depend on knowledge of the post, while this one probably did, and that is where I noted that I didn't follow the post.

Reply
[-]orthonormal30

I'm ashamed to admit this, but I haven't worked with programming on a deep enough level to be comfortable with your analogy. The one thing I think it's missing (and I haven't done a very good job explaining this) is the process of learning/introspection and the distinction between "adding to a body of propositional knowledge" and "triggering the 'learning' subroutine in the mind", which causes the central confusion.

Reply
[-]minusdash00

Propositional knowledge and introspection may be analogous to running a virtual machine in user-space, in which you can instantiate the redness object. But that's not a redness object in the real (non-virtual) program. The "real" running program only has user-space objects that are required for the execution of the virtual machine (virtual registers, command objects, etc).

Reply
[-]EniScien10

Serialization is a word that i needed to explain my own understanding. For not programmers, before i thought that it's how f(x)≠x, neural_notation(neural_structure)!=neural_structure. I We can't normally serialize our feelings, so if we get a "code" of red feeling, we can't find this structure in our brain and activate it to know not only neural structure, but it's inner feeling. I don't see something riddle in qualia, but i don't know how will being a bat with fully different brain, not with just unserializable new feeling channel.

Reply
[-]bogus90

And there it gets stuck. No new conscious connection formed between, say, "apple" and "red", and the learning agent is only looking for information in the conscious subgraph. The learning agent goes through a dozen false starts looking for something that isn't there. Martha articulates the thought of having learned something, but has nothing to say about what that new knowledge might be, except that it's something about the colors. This new knowledge is, somehow, ineffable to her.

Why do you state that Martha "has nothing to say" about her new knowledge? Here's an alternate possibility: the color pathways in her brain will light up, and her mental agents will start processing color information. As part of this processing, new nodes will appear in the 'conscious' semantic network corresponding to the principal components in said information, i.e. the main colors (including red); these nodes will subsequently collapse with the existing nodes for color words. As in the OP, Martha has the feeling of having learned something, but she can confidently state that she has "learned what red looks like".

This is in fact what happens when "extra senses" are added to the body by exploiting the adult brain's plasticity, such as with a magnetic belt (giving an "orientation sense") or by inserting small magnets under the skin.

Reply
[-]torekp80

Not only has she "learned what red looks like," but she has a memory of the experience and can refer to it, which she will verbally express to herself as, "so, red experiences are like that." And on the subvocalization of "that" her verbalizing agents will activate a chain that brings about the activation of that memory.

None of the larger points are affected by this; I just think it helps to get these details right.

Reply
[-]TimFreeman60

Excellent article. Walking through the scenario with these toy graph-based conscious entities substituted for the humans is a good first step that is likely to help with other conundrums like this.

So if this is what qualia are, any simulated human-like being that has both conscious and nontrivial unconscious processing has qualia, right?

Reply
[-]orthonormal40

So if this is what qualia are, any simulated human-like being that has both conscious and nontrivial unconscious processing has qualia, right?

It seems to me, right now, that it's also essential to have an agent which searches only the conscious graph for some purpose, in order to have something analogous to our qualia. I think I can imagine a mind that has a conscious/unconscious distinction but not that feature.

Reply
[-]thomblake40

So in short, "surprise" is not necessarily an indication of lack of "knowledge".

Reply
[-]orthonormal30

I prefer looking at it from the standpoint of "learning" rather than "surprise", but yes, the main idea is that we naturally equivocate between two meanings of "learning"- the phenomenon of adding to a body of knowledge, and the mental process that operates in us (most of the time appropriately) when this happens. Mary's Room just points out an edge case where the two meanings differ.

Reply
[-]beoShaffer20

Sounds cool any ideas on how we can test it experimentally? The theory fits with what I have already obsevered about quailia and does technically allow me to make predictions in advance, but all of the ones I can both think of and observe are ones I would have already been able to make based off of prior experience with qualia. There's a good chance thats just because I have a head ache and am not thinking to clearly, but I would still be interested in hearing if you have any predictions that you would care to make that aren't already clear from peoples intuitive understanding of qualia/other formal theories. I can thing of some new predictions regarding what I would expect to see if I directly obsevered the functioning of the human brain more closely than I am actually capable of.

Reply
[-]orthonormal10

First, welcome to Less Wrong!

That's a good question, and I haven't thought deeply enough about what novel predictions this model makes. Anyone else have suggestions?

Reply
[-]beoShaffer10

Ahh, novel predictions, it felt like there should be a more elegant way of saying that.

Reply
[-]AShepard20

I haven't read the post yet, but the title is awesome.

Reply
[-]AlephNeil90

A better title would be "nature red in truth and quale"

Reply
[-]Tyrrell_McAllister00

I'm not so sure. The familiar saying isn't "Nature, red in teeth and claws." It seems like there is a poetic convention of "mass-noun-ifying" nouns (if that's the right way to describe what's going on grammatically).

ETA: This remark was based on an error on my part.

Reply
[-]AlephNeil00

That may be right, but I don't see how it conflicts with my (throwaway) remark.

"Quale" works better than "qualia" because (i) it sounds more like the word "claw" and (ii) it's singular whereas 'qualia' is plural.

Reply
[-]Tyrrell_McAllister00

My remark was based on an error on my part. I forgot that qualia was the plural and quale was the singular, not the other way around.

Reply
[-]Tyrrell_McAllister10

orthonormal, is the following an accurate summary of your approach to the Mary's Room paradox?

Certain states of Mary's conscious mind cannot be brought about by any action of her conscious mind. In particular, no sequence of words can put her conscious mind into the state of experiencing red. But effable experiences are just those states of consciousness that can be induced by sequences of words. Therefore, Mary will find the experience of seeing red to be ineffable.

Reply
[-]orthonormal30

I feel that's inadequate, because it doesn't explain why (our subconscious mental model of) Mary feels she's learned something, when she knew in advance that her conscious mind would be put into a specific new state. That's why I think the key is to look at the mental agent that identifies learning. But I really need to rewrite this bit.

Reply
[-]timtyler10

One might suspect that this ability to model other minds is the real origin of the recent arms race in brain size among hominids.

Surely chimps do that too. The primary cognitive attribute that we have but chimps mostly don't is cultural inheritance. The picture there is one of apes with infected brains, whose brain cases swelled up to provide more room for their mutualist breathren. This is quite a different picture.

Reply
[-]JoshuaZ30

The primary cognitive attribute that we have but chimps mostly don't is cultural inheritance.

This seems wrong. Different chimp populations have different behavioral sets that seem learned from each other rather than genetic. See e.g. this summary. Humans have more effective methods of transmitting culture (especially through language), but chimps have quite a bit.

Reply
[-]timtyler10

Well, I did say "mostly". Human cultural inheritance and human language are leaps and bounds ahead of chimpanzee abilities in these areas.

In terms of culture, chimpanzees haven't made it as far as their "stone age" yet - and it wasn't until that point was reached that the human brain started inflating.

Reply
[-]orthonormal20

I hypothesize that, for whatever reason, there was a runaway selection in humans but not chimps for better modeling of others' behavior. Maybe our societies were more complicated, or less based on simple violent conflicts, than theirs. But of course, I only raised this as a suspicion.

I'm having a tough time seeing how the gene-level selection works in your account, though. Genes don't inherently care about memes, and larger cranial capacity comes with major disadvantages that require overwhelming genetic selective advantages to compensate.

Reply
[-]MixedNuts40

I thought that was the consensus.

Harry had once read a famous book called Chimpanzee Politics. The book had described how an adult chimpanzee named Luit had confronted the aging alpha, Yeroen, with the help of a young, recently matured chimpanzee named Nikkie. Nikkie had not intervened directly in the fights between Luit and Yeroen, but had prevented Yeroen's other supporters in the tribe from coming to his aid, distracting them whenever a confrontation developed between Luit and Yeroen. And in time Luit had won, and become the new alpha, with Nikkie as the second most powerful...

...though it hadn't taken very long after that for Nikkie to form an alliance with the defeated Yeroen, overthrow Luit, and become the new new alpha.

It really made you appreciate what millions of years of hominids trying to outwit each other - an evolutionary arms race without limit - had led to in the way of increased mental capacity.

'Cause, y'know, a human would have totally seen that one coming.

HPMoR, chapter 24

Reply
[-]timtyler40

That one is called the Machiavellian intelligence hypothesis - and it is covered in The Runaway Brain - and by Robin Dunbar in his "The Social Brain Hypothesis".

There are some other noteworthy ideas in the space as well - for instance the idea that omega-3 fats represented a nutritional constraint that got lifted by dietary changes. That hypothesis is laid out in The Driving Force.

No doubt all of these ideas have something to them - but the "memes did it" hypothesis is the one that gets my gold star these days.

Reply
[-]timtyler30

I'm having a tough time seeing how the gene-level selection works in your account, though. Genes don't inherently care about memes [...]

Memes have benefits to genes. Copying avoids the costs of individual learning. It allows you to obtain lots of good ideas quickly, by copying them from successful individuals. Good ideas are important. They can be mastery of fire, hunting techniques, love songs or negotiating strategies. Thus we have meme-spreading adaptations: our incessant babbling, our ultrasociality.

and larger cranial capacity comes with major disadvantages that require overwhelming genetic selective advantages to compensate.

These costs apply equally to any theory of human cranial expansion. Not much besides the enormous benefits afforded by cultural inheritance can reasonably expect to pay for them.

Certainly our big brains are the result of a sexually-selected arms race - but what was sexy was a GSOH, being able to sing love songs, etc. IOW, the products of cultural inheritance.

Reply
[-]PhilGoetz20

Citation needed. This is a specific question (do chimps model the minds of other chimps?) that cognitive scientists have considered in detail. But I don't remember specific studies or conclusions.

Reply
[-]Emile90

I don't have a reference ready, but I remember this from a book by Frans de Waal (probably "Chimpanzee Politics" - or maybe it was mentioned here):

Several primate species live in societies where only the Alpha male has reproduction rights - which doesn't mean the other males don't fool around when the Alpha isn't watching; and if the Alpha catches them, they will grovel and beg for mercy.

If a non-Alpha baboon mates while the Alpha is away, and then encounters the Alpha (who has no idea what happened), it will also grovel and beg for mercy - "Please don't hit me sir!". However, if a non-Alpha chimp does the same, when it encounters the Alpha it will pretend nothing happened - "Morning sir, hope all's fine sir!". That seems to indicate that chimps are better at modeling others than baboons are.

(some details are probably wrong, I'm not sure it was baboons)

Reply
[-]timtyler10

"Empathy" is a useful keyword here - so see, for example:

Chimps Not So Selfish: Comforting Behavior May Well Be Expression Of Empathy.

Empathy is pretty basic for social mammals - e.g.see: Dogs Probably Feel Sorry For Us.

Reply
[-]orthonormal00

I've just had some new insights, which may necessitate editing the post or even writing an additional one; I've started to think that the "Martha in the Mirror" section above is much weaker than the rest of my sequence, and I might have the idea now of how to fix that.

In the meantime, I've not received as much feedback (negative or positive) as I'd hoped, and I'm wondering why. If you could help me out with the following poll, I'd appreciate it...

(As usual, please only vote an option up, not down, so I can get an accurate tally. The last option is a karma equalizer for you to vote down so that I'm not grabbing free upvotes.)

Reply
[-]orthonormal60

POLL OPTION 1: The qualia sequence so far is a great success!

(Note the karma-equalizer option below.)

Reply
[-]orthonormal40

POLL OPTION 2: The qualia sequence so far adequately dissolves the Mary's Room argument, but doesn't adequately dissolve the experience of qualia.

(Note the karma-equalizer option below.)

Reply
[-]orthonormal30

POLL OPTION 3: The qualia sequence was superfluous; I thought, and still think, that qualia have been adequately dealt with before.

(Note the karma-equalizer option below.)

Reply
[-]orthonormal10

POLL OPTION 4: The qualia sequence is erroneous somewhere in a significant way. (Please comment and tell me where!)

(Note the karma-equalizer option below.)

Reply
[-]orthonormal00

POLL OPTION 6: My reaction doesn't fall into one of these categories, and I think you should include it as a poll option. (Please comment and specify!)

(Note the karma-equalizer option below.)

Reply
[-]orthonormal00

POLL OPTION 5: The qualia sequence so far is unclear to me, or poorly communicated. (If you can specify the locus of the problem, please do so!)

(Note the karma-equalizer option below.)

Reply
[-]sark00

If the learning agent does not find any new knowledge why does it make Martha report having learned something new? Why not make her feel as if nothing changed?

Reply
[-]orthonormal10

(Sort of redundant because we talked about this at the meetup, but:)

That's a contingent feature of the learning agent, not a necessary one. As with the feeling of déjà vu (searching memories for a match that isn't there), we're consciously aware of the error messages generated by this agent.

Reply
[-]Rob Bensinger-10

"[I]magine that we have created computational intelligence in the form of an autonomous agent that perceives its environment and has the capacity to reflect rationally on what it perceives. What would such a system be like? Would it have any concept of consciousness, or any related notions?

"To see that it might, note that one the most natural design such a system would surely have some concept of self — for instance, it would have the ability to distinguish itself from the rest of the world, and from other entities resembling it. It also seems reasonable that such a system would be able to access its own cognitive contents much more directly than it could those of others. If it had the capacity to reflect, it would presumably have a certain direct awareness of its own thought contents, and could reason about that fact. Furthermore, such a system would most naturally have direct access to perceptual information, much as our own cognitive system does.

"When we asked the system what perception was like, what would it say? Would it say, "It's not like anything"? Might it say, "Well, I know there is a red tricycle over there, but I have no idea how I know it. The information just appeared in my database"? Perhaps, but it seems unlikely. A system designed this way would be curiously indirect. It seems much more likely that it would say, "I know there is a red tricycle because I see it there." When we ask it in turn how it knows that it is seeing the tricycle, the answer would very likely be something along the lines of "I just see it."

"It would be an odd system that replied, "I know I see it because sensors 78-84 are activated in such-and-such a way." As Hofstadter (1979) points out, there is no need to give a system such detailed access to its low-level parts. Even Winograd's program SHRDLU (1972) did not have knowledge about the code it was written in, despite the fact that it could perceive a virtual world, make inferences about the world, and even justify its knowledge to a limited degree. Such extra knowledge would seem to be quite unnecessary, and would only complicate the processes of awareness and inference.

"Instead, it seems likely that such a system would have the same kind of attitude toward its perceptual contents as we do toward ours, with its knowledge of them being directed and unmediated, at least as far as the system is concerned. When we ask how it knows that it sees the red tricycle, an efficiently designed system would say, "I just see it!" When we ask how it knows that the tricycle is red, it would say the same sort of thing that we do: "It just looks red." If such a system were reflective, it might start wondering about how it is that things look red, and about why it is that red just is a particular way, and blue another. From the system's point of view it is just a brute fact that red looks one way, and blue another. Of course from our vantage point we know that this is just because red throws the system into one state, and blue throws it into another; but from the machine's point of view this does not help.

"As it reflected, it might start to wonder about the very fact that it seems to have some access to what it is thinking, and that it has a sense of self. A reflective machine that was designed to have direct access to the contents of its perception and thought might very soon start wondering about the mysteries of consciousness (Hofstadter 1985a gives a rich discussion of this idea): "Why is it that heat feels this way?"; "Why am I me, and not someone else?"; "I know my processes are just electronic circuits, but how does this explain my experience of thought and perception?"

"Of course, the speculation I have engaged in here is not to be taken too seriously, but it helps to bring out the naturalness of the fact that we judge and claim that we are conscious, given a reasonable design. It would be a strange kind of cognitive system that had no idea what we were talking about when we asked what it was like to be it. The fact that we think and talk about consciousness may be a consequence of very natural features of our design, just as it is with these systems. And certainly, in the explanation of why these systems think and talk as they do, we will never need to invoke full-fledged consciousness. Perhaps these systems are really conscious and perhaps they are not, but the explanation works independently of this fact. Any explanation of how these systems function can be given solely in computational terms. In such a case it is obvious that there is no room for a ghost in the machine to play an explanatory role.

"All this is to say (expanding on a claim in Chapter 1) that consciousness is surprising, but claims about consciousness are not. Although consciousness is a feature of the world that we would not predict from the physical facts, the things we say about consciousness are a garden-variety cognitive phenomenon. Somebody who knew enough about cognitive structure would immediately be able to predict the likelihood of utterances such as "I feel conscious, in a way that no physical object could be," or even Descartes's "Cogito ergo sum." In principle, some reductive explanation in terms of internal processes should render claims about consciousness no more deeply surprising than any other aspect of behavior. [...]

"At this point a natural thought has probably occurred to many readers, especially those of a reductionist bent: If one has explained why we say we are conscious, and why we judge that we are conscious, haven't we explained all that there is to be explained? Why not simply give up on the quest for a theory of consciousness, declaring consciousness itself a chimera? Even better, why not declare one's theory of why we judge that we are conscious to be a theory of consciousness in its own right? It might well be suggested that a theory of our judgments is all the theory of consciousness that we need. [...]

"This is surely the single most powerful argument for a reductive or eliminative view of consciousness. But it is not enough. [...] Explaining our judgments about consciousness does not come close to removing the mysteries of consciousness. Why? Because consciousness is itself an explanandum. The existence of God was arguably hypothesized largely in order to explain all sorts of evident facts about the world, such as its orderliness and its apparent design. When it turns out that an alternative hypothesis can explain the evidence just as well, then there is no need for the hypothesis of God. There is no separate phenomenon God that we can point to and say: that needs explaining. At best, there is indirect evidence. [...]

"But consciousness is not an explanatory construct, postulated to help explain behavior or events in the world. Rather, it is a brute explanandum, a phenomenon in its own right that is in need of explanation. It therefore does not matter if it turns out that consciousness is not required to do any work in explaining other phenomena. Our evidence for consciousness never lay with these other phenomena in the first place. Even if our judgments about consciousness are reductively explained, all this shows is that our judgments can be explained reductively. The mind-body problem is not that of explaining our judgments about consciousness. If it were, it would be a relatively trivial problem. Rather, the mind-body problem is that of explaining consciousness itself. If the judgments can be explained without explaining consciousness, then that is interesting and perhaps surprising, but it does not remove the mind-body problem.

"To take the line that explaining our judgments about consciousness is enough [...] is most naturally understood as an eliminativist position about consciousness [...]. As such it suffers from all the problems that eliminativism naturally faces. In particular, it denies the evidence of our own experience. This is the sort of thing that can only be done by a philosopher — or by someone else tying themselves in intellectual knots. Our experiences of red do not go away upon making such a denial. It is still like something to be us, and that is still something that needs explanation. [...]

"There is a certain intellectual appeal to the position that explaining phenomenal judgments is enough. It has the feel of a bold stroke that cleanly dissolves all the problems, leaving our confusion lying on the ground in front of us exposed for all to see. Yet it is the kind of "solution" that is satisfying only for about half a minute. When we stop to reflect, we realize that all we have done is to explain certain aspects of behavior. We have explained why we talk in certain ways, and why we are disposed to do so, but we have not remotely come to grips with the central problem, namely conscious experience itself. When thirty seconds are up, we find ourselves looking at a red rose, inhaling its fragrance, and wondering: "Why do I experience it like this?" And we realize that this explanation has nothing to say about the matter. [...]

"This line of argument is perhaps the most interesting that a reductionist or eliminativist can take — if I were a reductionist, I would be this sort of reductionist — but at the end of the day it suffers from the problem that all such positions face: it does not explain what needs to be explained. Tempting as this position is, it ends up failing to take the problem seriously. The puzzle of consciousness cannot be removed by such simple means."

—David Chalmers, The Conscious Mind: In Search of a Fundamental Theory (1996)

Reply
[-]TheAncientGeek-10

A shorter way of getting to the same conclusion.

If Orthonormal's argument summarises to

  1. Qualia have the property of ineffability, and no other. They are just cognitive blind spots.
  2. Ineffability can be explained reductionistically.
  3. Therefore, qualia can be explained reductionasitically.

Then the objection would be

  1. Qualia are crucial to morally relevance -- it is not OK to torture sims that actually feel pain.
  2. Ineffability, the possession of cognitive blind spots is not morally relevant. ("How dare you torture that sim, don't you realise it is unable to introspectively access everything in its mind!")
  3. Therefore, qualia are not defined solely by ineffability/inacessibility.
Reply
[-]anotheruser-20

The color red has many associations to the subconscious. These associations lie dormant and unused until the first time she actually experiences the color red. No amount of study could enable her to understand every nuance of these associations to the color red. I think that she really would learn something upon first seeing the color red.

Those things would probably not be particularly important because if they were someone would have taken the time to write them down at some point and Martha would have read about them. Things like "The color red is associated with aggression" are easy enough to learn but they are only qualitative, not quantitative. Until she actually experiences the color she will have no idea exactly how strong those associations are.

Perceiving a color has many very subtle effects on the mind that are not easy to identify even while you are experiencing them. When Martha first perceives the color red she experiences these connections for the first time. I think this would suffice to explain that Martha feels like she has learned something, because she actually has learned something, it's just too subtle to point out exactly what it is.

That said, I think you are right in principle in that different parts of the mind/ mental agents/ memes can work against each other by mistake or be harmful to the mind as a whole because they don't realize that they don't apply to a given situation. I just think that this particular scenario has a different explanation.

Reply
[-]MixedNuts20

She knows how the redness-aggression link works, she can compute it quantatively and know exactly how it would affect her behavior.

Reply
[-]anotheruser10

That is quite a lot of knowledge this Martha is supposed to have. If a human, or whatever species this hypothetical being Martha is, had so much knowledge about its own inner workings, would it really still be surprised? That Martha feels like she learned something new is by no means a given fact. We postulate that it would be so, based on the fact that we would think we had learned something new if we were in the same situation. If Martha really knows all this on a quantitative level, who are we to assume that she would still feel like she learned something?

That assumption is based on something we all have in common (our inability to understand ourselves in detail), but that would not be shared by Martha.

There is no way for us to know if such an intelligent and introspective being would actually learn something in this situation. This makes the question pointless if we assume that Martha is omnicient regarding her own psyche. The entire line of reasoning would depend on something which we can not actually know.

For this reason I was working under the assumption that Martha was merely extremely smart, compared to human scientists, but not able to analyze herself in ways we don't even begin to understand the implications of.

Reply
[-]Will_Sawin10

There's a limit to how much knowledge one can have about one's own inner workings since that knowledge would itself have inner workings. I think you can argue that nothing below the limit is sufficient to answer that problem.

Reply
[-]DSimon-10

You might be able to end this recursion problem ("I know about knowing things, but I don't yet know about knowing about knowing about things, and when I do know, then I'll have to go and learn to know about knowing about knowing about knowing...") by eventually reaching a point where each level is so similar to its predecessor that it and all its successors can be described with a quine.

Reply
[-]Will_Sawin10

The problem is that unpacking the quine requires computation, and you can experience the feeling of knowing from the results of a computation, e.g. by doing math.

Reply
Moderation Log
More from orthonormal
View more
Curated and popular this week
64Comments