EA's don't try to improve the wellbeing of their own group directly generally so they're reducing their own fitness. In nature, altruistic cabals form and split away from the less altruistic groups, and if altruism is an effective strategy for their adaptiveness then that altruistic community thrives.

That could be related to EA being a lot about signalling.Spending is a high value signal. Of course in principle it reduces your fitness in the long term. But being a costly signal is the whole point of signalling. So it keeps the size of the EA population small - which could be seen as the selection against altruism effect mentioned - but intended from a signalling point of view.