How's this for a metaphor: suppose I thought my mother had died in the Holocaust, when in fact she'd escaped the Holocaust without incident and simply lost contact with me. Someone makes Nazi jokes around me, or says that everyone who died in the Holocaust deserved it and went to Hell, or something equally offensive.

Suppose my interlocutor knows that my mother did not die in the Holocaust, and knows that if I believed my mother didn't die in the Holocaust I wouldn't be offended by what ey's saying. Ey also knows that since I do believe my mother died in the Holocaust, I definitely will be offended.

Even in this situation - in which I am only suffering because I have a false belief, and for reasons directly related to that false belief - I still think my interlocutor is very much in the wrong.