If anyone still has a hard time believing that this is not an unorthodox position among Philosophers, I'd like to recommend Shelly Kagan's excellent The Limits of Morality, which discusses 'radical consequentialism' and defends a similar conclusion.