In my experience, groups that want something to attack will attack groups that are generally aligned with them, rather than groups that are further away -- possibly due to the perceived threat of losing members to the similar group.

I've seen so many Communists get called Nazis by other Communist groups -- and those groups never go after people who actually call themselves Nazis.