I don't know how limited plasticity is. Speculation: maybe if we put on some color filter glasses that changes red with green or somehow mixes up the colors, then maybe even after a long time we'd still have the experience of the original red, even when looking at outside green material. Okay, let's say it's not plastic enough, we'd still feel an internal red qualia. But in what sense?

That would be an interesting experiment to do. We already know that people can adapt to wearing lenses that invert the picture or shift it laterally. Changing the colours while maintaining differences would be a little more complicated but quite feasible. You would need something similar to a VR headset, with a front-facing camera in front of each eye. The camera sensors would be connected, via some electronics to process the colours in any desired way, to the screen that each eye would see. This would be doable by a hobbyist with the necessary technical know-how. It might be as simple as cannibalising a couple of pocket cameras and switching some of the connections to the screen on the back.