The slander/libel case seems instructive: truth is an absolute defense against the accusation of slander or libel; it's the falsehood of a slanderous statement that harms.

Shouting the times-tables is a problem because of the delivery mechanism, not the content. Shouting anything at the top of your lungs for an hour in a public space is harmful to bystanders, and as you said, "offensive" is not what is wrong here.

"I fucked your mom", if true, is only potentially offensive for something like the following reasons:

• Swearing in polite company is frowned upon; "I had sex with your mother" is qualitatively different despite having the same content.
• It's an implication of promiscuity (or low selectiveness of sexual partners) on the part of the target's mother, and our society's views on sexuality derogate promiscuity, turning this empirical statement into an insult. Arguably, this is a problem with society's views on sexuality ("slut shaming"), rather than the fact that informing someone about their sexual encounters with that person's mother is inherently offensive.

In short, I don't think I buy your claim that "Some empirical statements, orthogonal to truth or falsity, are offensive." At least, I'd like to see it supported better before I consider it. This isn't simply contrarianism; I think that the ability and right to say true things regardless of whether someone finds those truths unpleasant is extremely important, and social norms to the contrary should not be adopted or perpetuated lightly.