I see, thanks. I was thinking of "to" as just being the other direction of "by", but you can interpret it as more like "towards" and then it's all good.

Oh, I bet it's because the previous sentence was "When someone draws Mohammed, it is considered offensive to Muslims.", and that one seemed like a straightforward "no-one except Muslims is being offended by this" mapping, which was then extended to cover sexism.