Rich people do not harm the poor with their wealth: they simply may not help (aside from, say, putting their money into banks and loaning it out to less wealthy people).

Significant status differences in a society are correlated with all kinds of adverse outcomes. One causal hypothesis (with quite a bit of compelling evidence backing it) is that a lot of this has to do with the neuroendocrinological stress response triggered by the perception that others are higher status than oneself.

I don't know if I'd classify this as rich people harming poor people, but (if accurate) it is an example of entrenched social inequality harming poor (and other low-status) people.