If we postulate that physicalism is true, then there are no qualia by definition

But I'm not postulating that physicalism is true. (Although it looks to me like it probably is.) I am assuming only that physics gives a correct account of what brains and bodies do. (Or, as I put it before, your brain runs on physics and not on magic.) My understanding is that e.g. David Chalmers does not disagree with this. Of course if some kind of substance dualism is true -- if some of what I think your brain does is actually being done by an immaterial soul that interfaces with your brain at the pineal gland, or something -- then there can be qualia differences without physical differences; that idea has its own problems but they are quite different ones, and the argument against that view has a very different shape.

my empirical experiences contradict my best theory of reality

How?

Your empirical experiences show that there is such a thing as an experience of seeing red. They do not show that this experience isn't built on top of physics. They do not show that your experience of seeing red could be detached from all the physical phenomena involved in actually seeing red things. It's ideas like those that contradict your best theory of reality, and those ideas are not simply a matter of observing your empirical experiences.

(At least, I can't imagine how they could be. But perhaps your empirical experiences are spectacularly unlike mine.)

You are arguing that if all associations are the same, the experience must be the same

No. I am arguing that if the physical causes are the same, and the associations are the same, and everything else we can actually observe is the same, then we cannot have good reason to think that the experience is different; and that, in view of all the evidence favouring physicalism, much the simplest and most plausible conclusion is that in fact the experience is the same when all physical things are the same.

I could imagine my self looking of large red field and looking on large green field, without any associations about them and still having different experiences of their colour.

I agree. (Except that someone who really had no associations with either could hardly actually be your self or mine.) But what's required for zombies or "inverted spectrum" is quite different and goes much further.

We need, let's say, three Turchins. I'll call them R, RG, and G. R and G are versions of real-world-you; R is looking at a field of poppy flowers and G is looking at a field of grass. RG is different in some mysterious way; he is looking at a field of poppy flowers just like R, having the exact same experience as G is having, but also having the exact same behaviour as R. And whatever happens in the physical world, RG's experience is going to continue to match G's and his behaviour is going to continue to match R's.

Now, notice that this scenario as I've stated it is actually extra-specially ridiculous when taken seriously. For instance, suppose you walk with RG into the poppy field and ask about the shape of the plants he's looking at. For his behaviour to be an exact match for G's, he will need to experience seeing grass stalks and leaves while telling you about the poppy plants he sees. This is, for me, firmly in "lizardmen in the White House" territory.

It is to avoid this kind of ridiculousness that people advocating this kind of detachment of qualia from physics always pick on a very simple kind of quale, one with scarcely any "dimensions" to it, namely colour. No one ever talks about how maybe turchinB looks at a square, has the experience of looking at a hexagon, but behaves exactly as if he sees a square (including e.g. counting off its vertices correctly). No one ever talks about how maybe turchinB watches Casablanca, has the experience of watching The Matrix, but behaves exactly as if he is watching Casablanca. No one ever talks about how maybe turchinB eats a really hot Thai curry, has the experience of eating a rather bland lasagne, but behaves exactly as if he is eating a curry. Because the absurdity of those scenarios is plain to see: these are complex experiences, with parts that interrelate, and probing them more closely gives results that are preposterous if we imagine them interchanged. But colour? Well, colour's pretty simple; perhaps seeing red and seeing green could be interchanged. ... But then I say "hold on, actually colour isn't so simple because there are all these other things you associate with different colours, and these physiological things that happen when you see red, etc.", and of course the response is to say: yeah, OK, so let's imagine some kind of stripped-down version of seeing red where somehow all those associations aren't there, and then maybe they could be interchanged.

In other words, if you ignore almost all kinds of experience and focus on one particularly simple one, and then make the further simplification of assuming that all the complications even that simple kind of experience actually has ... why, then you can imagine (or at least you say you can imagine) two different experiences of this kind being interchanged without physical consequence.

I say that what you're trying to imagine is so utterly divorced from reality that thinking you can imagine it tells us nothing about how the world actually is. And that once you're considering things so simple, it becomes much harder to resist the reductionist argument: "We can, in principle, trace the physical consequences of seeing a red thing; we can see neural circuits that get activated, etc.; we can see how this differs from seeing a green thing. Why should we think this bare red-or-green quale, with no further richness to it, no associations or consequences or anything, is anything other than the activation of one or another set of neurons?"

(For more interesting qualia, I can feel the attraction of saying that they surely must be something more than neuron activations -- though I think that's actually a mistake. "How could a mere adjustment of electrical potentials and ion concentrations be so thrilling/moving/terrifying/...?" But for me, at least, that intuition entirely goes away when I consider the stripped-down minimal qualia that seem to be the only ones for which the sort of "inversion" you're talking about seems remotely plausible.)