Two points:

One. Charity, up to a point, is not necessarily a trade-off. Just as adding a hobby can make you more productive at work by forcing you to be efficient with your time, adding a charitable commitment can force you to stop wasting money. There is a reason why the Judaeo-Christian tradition recommends tithing; a tenth of income is a good rule of thumb for an amount that's significant but not enough to make you noticeably poorer.

Two. When people have personal problems as a result of altruism, I suspect it's the nature of the charity (futurist ideas sound useless to a lot of people) or the nature of the commitment (giving more than a tenth of income, for example) or some interpersonal issue that the altruist doesn't understand. I want to emphasize that last possibility. If you know you have Asperger's, you should be extra skeptical about your own ability to explain interpersonal behavior.