If that's how you feel, replace it with "greedy Jewish bankers". Real ones exist, but it's unlikely that someone complaining about them is limiting his complaints to the ones that exist--he's probably saying it because he thinks there are more greedy Jewish bankers than there really are.

The point is that it may make sense to object to an otherwise legitimate attack on a group that doesn't include you if the attacker thinks that the group includes you.