"I feel confident being upset at people who draw swastikas where they expect Jews to see them, or burn crosses outside the houses of black people."

The vileness of the swastika doesn't come form the subjective reactions of Jews who see it. A swastika is an implicit call to genocide. I think you are far too hung up on what are relatively insignificant subjective consequences, to the point of ignoring the overarching political significance of the acts in question. (Such microscopy is indeed Dr. Hanson's method.) Depending on context drawing a picture of Mohammed can be (among other things) a call to persecution of persons professing Islam, an objection to Islamic censorship, or serve some purely artistic purpose, each having consequences that far outweighs "offensiveness."