"Qualia" is just a new word for what used to be meant by the word "sensations", before "sensation" was redefined to mean "a type of brain process". The idea that sensory qualities like color are in the sensations, and hence in us, has been around for hundreds of years - thousands, if you count Democritus.
The problem with the modern redefinition of "sensation" as "brain process" is now that color is nowhere at all, inside or outside the brain. Or, more precisely, it substitutes a particular theory of what a sensation is (brain event) for the thing itself (experience of a sensory quality) in a way which allows the latter to be ignored or even denied.
On this issue most materialists are dualists - property dualists - without even noticing it. The problem is very simple. Physics, and hence natural science, is based on a model of the world in which all that exists are fundamental entities (particles, wavefunctions, etc) which do not possess the "secondary sensory qualities" like color, either individually or in combination. There is a disjunction between the properties posited by physics and the properties known in experience.
There are three known ways of dealing with this while still believing in physics. You say that both properties exist - property dualism. You say that only physical properties exist - total "eliminativism". Or you say that the experiential properties are directly playing a role in physics - which is best known via panpsychism, but one might doubt the necessity to regard everything ("pan") as "mental" ("psych").
Most materialists are property dualists because they say that only atoms exist, but then they think of their experience as how it feels to be a particular arrangement of atoms, when there is no such property in physics. It's an extra thing being tacked onto the physics. And the realization that this is dualism is somehow pushed away by the use of a locution like "experiencing blue" - e.g. "my current state includes the property that I am experiencing blue" - which buries the fact that the sensation itself has the property of being blue.
It's the fact that something is blue, which is why "qualia" have to be "posited".
It's only really your second paragraph that I disagree with. I'm a panpsychist, but I don't often mention it because a lot of people take that to mean "I believe that everything in the universe has a mind, including rocks and stars".
I go even further than you, though. I think that even of the materialists who aren't accidental/secret property dualists, most of them are still dualists without realising it. The idea that there are physical objects which are related to one another causally is inherently dualist because it theorises two types ...
[Cross-posted.]
1. Defining the problem: The inverted spectrum
A. Attempted solutions to the inverted spectrum.
B. The “substitution bias” of solving the “easy problem of consciousness” instead of the “hard problem.”
2. The false intuition of direct awareness
A. Our sense that the existence of raw experience is self-evident doesn’t show that it is true.
B. Experience can’t reveal the error in the intuition that raw experience exists.
C. We can’t capture the ineffable core of raw experience with language because there’s really nothing there.
D. We believe raw experience exists without detecting it.
3. The conceptual economy of qualia nihilism pays off in philosophical progress
4. Relying on the brute force of an intuition is rationally specious.
Against these considerations, the only argument for retaining raw experience in our ontology is the sheer strength of everyone’s belief in its existence. How much weight should we attach to a strong belief whose validity we can't check? None. Beliefs ordinarily earn a presumption of truth from the absence of empirical challenge, but when empirical challenge is impossible in principle, the belief deserves no confidence.