This post grew out of a conversation with Laurent Orseau; we were initially going to write a paper for a consciousness/philosophy journal of some sort, but that now seems unlikely, so I thought I'd post the key ideas here.

A summary of this post can be found here - it even has some diagrams.

The central idea is that thinking in terms of AI or similar artificial agent, we can get some interesting solutions to old philosophical problems, such as the Mary's room/knowledge problem. In essence, simple agents exhibit similar features to Mary in the thought experiments, so (most) explanations of Mary's experience must also apply to simple artificial agents.

To summarise:

• Artificial agents can treat certain inputs as if the input were different from mere information.
• This analogises loosely to how humans "experience" certain things.
• If the agent is a more limited (and more realistic) design, this analogy can get closer.
• There is an artificial version of Mary, mAIry, which would plausibly have something similar to what Mary experiences within the thought experiment.

Edit: See also orthonormal's sequence here.

Mary's Room and the Knowledge problem

In this thought experiment, Mary has been confined to a grey room from birth, exploring the outside world only through a black-and-white monitor.

Though isolated, Mary is a brilliant scientist, and has learnt all there is to know about light, the eye, colour theory, human perception, and human psychology. It would seem that she has all possible knowledge that there could be about colour, despite having never seen it.

Then one day she gets out of her room, and says "wow, so that's what purple looks like!".

Has she learnt anything new here? If not, what is her exclamation about? If so, what is this knowledge - Mary was supposed to know everything there was to know about colour already?

Incidentally, I chose "purple" as the colour Mary would see, as the two colours most often used, red and blue, lead to the confusion as to what "seeing red/blue" means - is this about the brain, or is it about the cones in the eye. But seeing purple is strictly about perception in the brain.

Example in practice

Interestingly, there are real example of Mary's room-like situations. Some people with red-green colour-blindness can suddenly start seeing new colours with the right glasses. Apparently this happens because the red and green cones in their eyes are almost identical, so tend to always fire together. But "almost" is not "exactly", and the glasses force green and red colours apart, so the red and green cones start firing separately, allowing the colour blind to see or distinguish new colours.

Can you feel my pain? The AI's reward channel

This argument was initially presented here.

AIXI

Let's start with the least human AI we can imagine: AIXI, which is more an equation than an agent. Because we'll be imagining multiple agents, let's pick any computable version of AIXI, such as AIXItl.

There will be two such AIXItl's, called $A_r$ and $A_q$, and they will share observations and rewards: at turn $i$, this will be $o_i$, $r_i$, and $q_i$, with $r_i$ the reward of $A_r$ and $q_i$ the reward of $A_q$.

To simplify, we'll ignore the game theory between the agents; each agent will treat the other as part of the environment and attempt to maximise their reward around this constraint.

Then it's clear that, even though $r_i$ and $q_i$ are both part of each agent's observation, each agent will treat their own reward in a special way. Their actions are geared to increasing their own reward; $A_r$ might find $q_i$ informative, but has no use for it beyond that.

For example, $A_r$ might sacrifice current $r_i$ to get information that could lead it to increase $r_{j>i}$; it would never do so to increase $q_{j>i}$. It would sacrifice all $q$-rewards to increase the expected sum of $r_i$; indeed it would sacrifice its knowledge of $q_i$ entirely to increase that expected sum by the tiniest amount. And $A_q$ would be in the exact opposite situation.

The $A_r$ agent would also do other things, like sacrificing $r_i$ in counterfactual universes to increase $r_i$ in this one. It would also refuse the following trade: perfect knowledge of the ideal policy that would have maximised expected $r_i$, in exchange for the $r_i$ being set to $0$ from then on. In other words, it won't trade $r_i$ for perfect information about $r_i$.

So what are these reward channels to these agents? It would go too far to call them qualia, but they do seem to have some features of pleasure/pain in humans. We don't feel the pleasure and pain of others in the same way we feel our own. We don't feel counterfactual pain as we feel real pain; and we certainly wouldn't agree to suffer maximal pain in exchange for knowing how we could have otherwise felt maximal pleasure. Pleasure and pain can motivate us to action in ways that few other things can: we don't treat them as pure information.

Similarly, the $A_r$ doesn't treat $r_i$ purely as information either. To stretch the definition of a word, we might say that $A_r$ is experiencing $r_i$ in ways that it doesn't experience $q_i$ or $o_i$.

Let's try and move towards a more human-like agent.

TD-Lambda learning

TD stands for temporal difference learning: learning by the difference between a predicted reward and the actual reward. For the TD-Lambda algorithm, the agent uses $V\left(s\right)$: the estimated value of the state $s$. It then goes on its merry way, and as it observes histories of the form $\dots s_{i-1}{a}_{i-1}{r}_{i-1}{s}_i{a}_i{r}_i{s}_{i+1}{a}_{i+1}{r}_{i+1}$, it updates is estimate of all its past $V\left(s_i\right)$ (with a discount factor of $0\le \lambda \le 1$ for more distant past states $s_{j<i}$).

Again, imagine there are two agents, $T_r$ and $T_q$, with separate reward functions $r$ and $q$, and that each agent gets to see the other's reward.

What happens when $T_r$ encounters an unexpectedly large or small value of $q_i$? Well, how would it interpret the $q_i$ in the first place? Maybe as part of the state-data $s_{i+1}$. In that case, an unexpected $q_i$ moves $T_r$ to a new, potentially unusual state $s_{i+1}$, rather than an expected $s_{i+1}^{\prime }$. But this is only relevant if $V\left(s_{i+1}\right)$ is very different from $V\left(s_{i+1}^{\prime }\right)$: in other words, unexpected $q_i$ are only relevant if they imply something about expected $r_i$. And even when they do, their immediate impact is rather small: a different state reached.

Compare what happens when $T_r$ encounters an unexpectedly large or small value of $r_i$. The impact of that is immediate: the information percolates backwards, updating all the $V\left(s_{j<i}\right)$. There is an immediate change to the inner variables all across the agent's brain.

In this case, the 'experience' of the $T_r$ agent encountering high/low $r_i$ resembles our own experience of extreme pleasure/pain: immediate involuntary re-wiring and change of estimates through a significant part of our brain.

We could even give $T_r$ a certain way of 'knowing' that high/low $r_i$ might be incoming; maybe there's a reliability score for $V\left(s_i\right)$, or some way of tracking variance in the estimate. Then a low reliability or high variance score could indicate to the $T_r$ that high/low $r_i$ might happen (maybe these could feed into the learning rate $\alpha$). But, even if the magnitude of the $r_i$ is not unexpected, it will still cause changes across all the previous estimates - even if these changes are in some sense expected.

mAIry in its room

So we've established that artificial agents can treat certain classes of inputs in a special way, "experiencing" their data (for lack of a better word) in a way that is different from simple information. And sometimes these inputs can strongly rewire the agent's brain/variable values.

Let's now turn back to the initial thought experiment, and posit that we have a mAIry, an AI version of Mary, similarly brought up without the colour purple. mAIry stores knowledge as weights in a neural net, rather than connections of neurons, but otherwise the thought experiment is very similar.

mAIry knows everything about light, cameras, and how neural nets interpret concepts, including colour. It knows that, for example, "seeing purple" corresponds to a certain pattern of activation in the neural net. We'll simplify, and just say that there's a certain node $n_p$ such that, if its activation reaches a certain threshold, the net has "seen purple". mAIry is aware of this fact, and can identify the $n_p$ node within itself, and perfectly predict the sequence of stimuli that could activate it.

If mAIry is still a learning agent, then seeing a new stimuli for the first time is likely to cause a lot of changes in the weights in its nodes; again, these are changes that mAIry can estimate and predict. Let $c_p$ be a Boolean corresponding to whether these changes have happened or not.

What dreams of purple may come...

A sufficiently smart mAIry might be able to force itself to "experience" seeing purple, without ever having seen it. If it has full self-modification powers, it could manually activate $n_p$ and cause the changes that result in $c_p$ being true. With more minor abilities, it could trigger some low-level neurons that caused a similar change in its neural net.

In terms of the human Mary, these would correspond to something like self-brain surgery and self-hypnosis (or maybe self-induced dreams of purple).

Coming out of the room: the conclusion

So now assume that mAIry exits the room for the first time, and sees something purple. It's possible that mAIry has successfully self-modified to activate $n_p$ and set $c_p$ to true. In that case, upon seeing something purple, mAIry gets no extra information, no extra knowledge, and nothing happens in its brain that could correspond to a "wow".

But what if mAIry has not been able to self-modify? Then upon seeing a purple flower, the node $n_p$ is strongly activated for the first time, and a whole series of weight changes flow across mAIry's brain, making $c_p$ true.

That is the "wow" moment for mAIry. Both mAIry and Mary have experienced something; something they both perfectly predicted ahead of time, but something that neither could trigger ahead of time, nor prevent from happening when they did see something purple. The novel activation of $n_p$ and the changes labelled by $c_p$ were both predictable and unavoidable for a smart mAIry without self-modification abilities.

At this point the analogy I'm trying to draw should be clear: activating $n_p$ and the unavoidable changes in the weights that causes $c_p$ to be true, are similar to what a TD-Lambda agent goes through when encountering unexpectedly high or low rewards. They are a "mental experience", unprecedented for the agent even if entirely predictable.

But they are not evidence for epiphenomenalism or against physicalism - unless we want to posit that mAIry is non-physical or epiphenomenal.

It is interesting, though, that this argument suggests that qualia are very real, and distinct from pure information, though still entirely physical.