While the term is clearly overused, I think it is not entirely useless. One useful method is to focus not on the victim but on the speaker. Does the speaker sound like someone bringing others down just to feel better about himself? For example, Murray's The Bell Curve was attacked a lot for its alleged racism, but it was not actually hate speech, the tone was that of a deeply concerned and frightened and reluctant admittance of human differences. However just go on reddit and you find many examples of people engaging in cheap potshots of racism or sexism largely as a way to comparatively feel better about themselves. If a racial or gender oriented comment sounds like something said by a gamma male in a proverbial parental basement to feel comparatively better, it is hate speech. /r/fatpeoplehate is a good example, if you count the various kinds of insults "butter slugs" "butter golems" etc.

If it sounds strange, then let me remind you that hate is an emotion. It is something felt by people. So hate depends on not whether it had or had not hurt the victim, but whether the perpetrator felt it or not. So offensive speech (victim angle) can still not be hate speech (perp angle) and vice versa.