I basically agree with what you said, but I would go one step further.

That is, we don't learn about redness as such. When we get the sensory stimuli of redness, we aren't learning the ability to recognize red objects as such. What we are learning is things and causes in the world and redness is just one attribute of the sensory data. It is useful, because it makes the prediction of things in the world more accurate.

Let's take one type of cause in the world that we learn: objects. When we are exposed to visual stimuli of a certain type of object, we learn to expect it to have some type of common pattern. One aspect of this is the color of the object. If the color is nearly always red, we learn to predict that this type of object is typically red. So our prediction of this type of cause in the world is enhanced by recognizing this attribute of the sensory data.

If the sensory data lacks color completely, or if the color is not what we expect it to be, the prediction of our brain is not as accurate. But we might be able to still make the correct inference of the world, it's just a bit harder. If the color is completely different of what we've learned it to be, we are typically still able to recognize the object, but we are surprised about the color, because it doesn't match our representation. Our attention is drawn to the peculiar color.

The high level representation of the object of certain type, then, is composed of these different aspects of what we have seen before. These different aspects are things like the pattern of the visual stimuli, the intensity of the visual signal, the color of the signal, etc. A person who has been exposed to color stimuli all her life, has these extra bits of information about objects of the world she has learned about. So "sensation of red" isn't a thing or place in the brain. It's not even a process in the brain. Redness (and color stimuli in general) is part of our knowledge of the world. It's part of the objects and categories we have learned. It's part of the representation of these objects in our brain.

So when Mary, in her black and white room, tries to learn about colors, she cannot directly influence her representation of the world in her brain. She would have to twiddle every piece of knowledge in her head with regards of its color.

As an example, let's say she has seen a lot of roses from her black and white TV. (Let's assume for simplicity that roses were pretty much always red.) Now, from her sources, she learns that roses are typically red. She would then have to locate the representation of roses in her head and add a probabilistic adjustment if the color of the visual stimuli is red. It's something like: "if the visual pattern is like this and if the color is red, then it's likely that's it's a rose." Or more precisely, it's some sort of conditional probabilistic table.

So for Mary to learn redness before being exposed to color stimuli, she would have to modify all the bits of knowledge of the world that she has acquired through visual data. This would mean all types of objects and other visual information she has. She would have to add this conditional clause to every type of object she has learned where the color red has any significance to the inference of the object. She would have to add this attribute also to every specific knowledge of the world she has too.

So in summary, this thought experiment seems to me like a huge misunderstanding of how we learn about the world.