It certainly matters to the subject! I sure wouldn't want to lose my ability to experience regardless of whether others can ever notice (or whether it's possible).

Your objection here strikes me a little bit like behaviorism. Yes, there are valuable things to be gotten most of the time from such an approach, but behaviorism suffered from an unwillingness to acknowledge that people had thoughts. After all, thoughts didn't demonstrate themselves in behavior beyond being talked about, in which case it was the talk that was part of the scientific domain, not the thinking. The thing is, I know I think, and my strong impression is that others talk about "thoughts" for the same reason I do: they think. The fact that behaviorism didn't have a clear empirical approach to exploring these subjective experiences tagged "thoughts" didn't mean that they were uninteresting or irrelevant. This was the main reason why we switched away from behaviorism in the second half of the 20th century.

(Quick note: Yes, I know that not all behaviorism was like this. Some behaviorists simply said that they didn't want to make claims about what thinking entailed because they didn't know how to approach the matter empirically. However, it was common if not dominant among behaviorists to take the "If we can't study it then it doesn't exist or doesn't matter" approach.)

In exactly the same way, quale-type experience seems to be present for me, and my own impression is that everything I talk about in terms of empiricism gets filtered through qualia. There are no data that transmit information to my mind that I am aware of without that awareness taking on qualia. I'm under the strong impression that others experience the world similarly. The fact that we don't know what that means in reductionistic terms doesn't mean that it's irrelevant or that qualia don't exist. It just means that we don't know how to approach the question unambiguously as yet.