I think you are confusing the word "color" that identifies a certain type of visual experience, with the word "color" that identifies a certain set of light-frequencies. This is much like confusing the word "sound" which means "auditory experience", with the word "sound" which means "acoustic vibrations".
You see certain frequencies in a different way than people with red-green colour blindness; in short these frequencies lead to different qualia, different visual experiences. That's rather obvious and rather useless in discussing the deeper philosophical point.
But to say that you experience certain visual experiences differently than others experience them, may even be a contradiction in terms -- unless it's meant that the atomic qualia trigger in turn different qualia (e.g. different memories or feelings) in each person. Which is probably also trivially true...
Apologies for the confusion.
Your second paragraph encapsulates the point I intended to convey; that given frequencies of light create in my mind qualia that differ from the qualia created by the same frequency of light in the mind of a red-green colourblind person.
[Cross-posted.]
1. Defining the problem: The inverted spectrum
A. Attempted solutions to the inverted spectrum.
B. The “substitution bias” of solving the “easy problem of consciousness” instead of the “hard problem.”
2. The false intuition of direct awareness
A. Our sense that the existence of raw experience is self-evident doesn’t show that it is true.
B. Experience can’t reveal the error in the intuition that raw experience exists.
C. We can’t capture the ineffable core of raw experience with language because there’s really nothing there.
D. We believe raw experience exists without detecting it.
3. The conceptual economy of qualia nihilism pays off in philosophical progress
4. Relying on the brute force of an intuition is rationally specious.
Against these considerations, the only argument for retaining raw experience in our ontology is the sheer strength of everyone’s belief in its existence. How much weight should we attach to a strong belief whose validity we can't check? None. Beliefs ordinarily earn a presumption of truth from the absence of empirical challenge, but when empirical challenge is impossible in principle, the belief deserves no confidence.