This "Mary's Room" argument, like the "Chinese Room" argument†, contains a subtle sleight of hand.
On the one hand, for the learning to be about just the qualia rather than about externally observable features of vision processing, the subject would need to learn immensely more than the physical properties of red light. (The standard version of Mary's Room does so, postulating Mary to also deeply understand her own visual cortex and the changes it would undergo upon being exposed to that color.) In fact, the depth of conscious theoretical understanding that this would require is far beyond any human being, and it's wrong and silly to naively map our mind-states onto those of such a mind.
On the other hand, it plays on the everyday intuition that if I've never seen the color red, but have been given a short list of facts about it and am consciously representing my limited intuition for that set of facts, that doesn't add up to the experience of seeing red.
The equivocation consists of thinking that a superhuman level of detailed understanding of (and capability to predict) the human brain can be analogized to that everyday intuition, rather than being unimaginably other to it. So I don't see that an agent who was really possessed of that level of self-understanding would necessary feel that the actual experience added an ineffable otherness to what they already knew.
That sense of ineffable otherness, IMO, comes from the levels of detail in the mental processing of color which we don't have conscious access to. Our conscious mind isn't built to understand what we're doing when we visually perceive, at the level that we actually do it-- there's no evolutionary need to communicate all the richness of color perception, so the conscious mind didn't evolve to encompass it all. And this limitation of our conscious understanding feels to us like a thing we have which cannot in principle be reduced.
† The application of this same principle to the Chinese Room argument is a trivial exercise, left to the reader.
Intuitions don't matter. If Mary can't activate her neural pathways participating in creation of experience of seeing red, then she has no means of knowing how she will experience redness. All models she can create in her mind will be external to her as the mind created by actions of human being in Chinese room is external to that human being.
And this limitation of our conscious understanding feels to us like a thing we have which cannot in principle be reduced.
It is not only conscious understanding that is required, we will need a conscious control of...
This is our monthly thread for collecting these little gems and pearls of wisdom, rationality-related quotes you've seen recently, or had stored in your quotesfile for ages, and which might be handy to link to in one of our discussions.