This thought experiment always seemed silly to me. As if somehow the experience of the visual cortex reacting to "color" input was not a piece of knowledge.

If someone has a poor ability to mentally visualize 3-dimensional objects, and is shown a set of formula that will draw a specific and very odd object (learning everything but what the object actually looks like), and is only ever allowed to graph on paper, then of course when we finally hand them a physical model of the object we have given them new information.

I don't see this as any different. We have imposed some limitation on a subject, given them every piece of knowledge that the limitation allows for, and then called this "all knowledge" (which it clearly is not). After you remove the limitation and the final pieces of knowledge are gained, you have not demonstrated that non-physical knowledge exists. The fallacy was calling the knowledge given to the restricted subject complete, when it was in fact not.

The knowledge is only new because of a PHYSICAL limitation that previously existed. Once the PHYSICAL limitation was removed, a PHYSICAL interaction resulted in new PHYSICAL knowledge.

All you have demonstrated is that it is possible restrict knowledge.