I often agree that things labeled evil do more harm than good, but if the person doing the 'evil' thing agreed, they wouldn't be doing it, so it's obviously at least somewhat debatable.
They cause harm to you, and good to the person doing it. Nothing to disagree about.
The discussions in question have generally been about the actions of third-parties in other parts of the world, which haven't had any appreciable effect on my life (unless you count 'taking thought-time away from other issues' as an effect).
In cases where the discussion is about something that's been done to me, I still don't use the word 'evil', and I've actually been known to object to other people doing so in those cases. 'Selfish', 'misguided', 'poorly informed', 'emotion driven', and the like cover those situations much more usefully.
Tyler Cowen argues in a TED talk (~15 min) that stories pervade our mental lives. He thinks they are a major source of cognitive biases and, on the margin, we should be more suspicious of them - especially simple stories. Here's an interesting quote about the meta-level: