This is an interesting idea, to be sure. It sounds like you might be sympathetic to virtue ethics (as indeed am I, though I become a consequentialist when the stakes get high).

Also, have you read Bernard Williams' critique of utilitarianism? Because

You're free to choose those values for yourself -- you don't have to drop them because they're perhaps not optimal for the world's well-being.

would definitely be one of his arguments.