You've pinpointed it: the only difference is who gets it. When investing, diversification as the receiver of the return is useful because you'd rather gain slightly less than often lose everything. When ... living, diversification as the receiver of the return is useful for the same reason.

When investing, you'd like your buyers to diversify... but there's only one buyer, so that buyer needs to diversify. But when giving charitably, the world would like its buyers to diversify, and there are lots of buyers. Assuming its buyers are sufficiently independent, the world gets enough diversification just because its buyers make different decisions. So as long as sufficiently many people make different charitable giving decisions than you, feel free to buy only what you think are the most efficient charities.

The world doesn't care how much you help it, the world only cares how much it gets helped overall.