Well, “X is offensive” is not something I'd normally say -- I'd specify who is offended (e.g. “I'm offended by X”, or “X might offend [class of people]”), even though sometimes “[class of people]” is as generic as “someone”.

Fair enough, but it's not obvious that the mere fact of someone being offended is something I (or "we") should care about.

I noted here that

[W]hether we as a society agree that the target is entitled to take offense seems like the straightforward operationalization of implementing the two-place function of offense as a one-place function. So when I say "I don't think X should be considered offensive", I'm not making any sort of claim about whether any particular person will in fact take offense; the claim I am making is something along the lines of "we should not consider offense taken at X to be justified, and we should not care about said offense, or modify our behavior (i.e. stop saying X) on the basis of said offense".

As for fixing the listener's deficiencies...

You mean in principle or in practice?

Well, here's the thing. Let's say I say something to someone, or a group of someones, that this person(s) finds offensive. Let's say it's the case that in principle, the situation would be fixed (that is, the offense obviated) by suitably "fixing" the listener, but in practice this is not feasible.

The question still remains: did I do anything wrong? If so, what?

Well, I might plausibly be guilty of not knowing my audience. That's an important skill to have and use. Some people, though, seem to behave as though any instance of a speaker saying something that is offensive to anyone who (by intent or otherwise) hears it, constitutes a horrible crime on the part of the speaker, and not only is inherently terrible, but reveals personal evil.

And my response is: no, if this offense would not have happened but for the listener's stupidity or insanity, then all that's happened here is that the speaker might have to exercise more caution on what to say to whom. We should not throw our social approval behind the listener's offense (which is what we seem to mean in practice when we label utterances as "offensive"). We should not demand groveling public apologies, excoriate the speaker for being a terrible person, demand that he/she never say such things again, kick him out of our club, demand that policies be put in place to prevent such horrible things from being said ever again, etc. etc.

Because there's always going to be someone who is sufficiently stupid or insane to be offended by virtually anything. And when that "anything" happens to be the truth, then by socially approving the offense taken, we create an environment where the truth (even if it's only a specific subset of the truth) is less likely to be spoken. That is a great loss.