I've previously dwelt in considerable length upon forms of rationalization whereby our beliefs appear to match the evidence much more strongly than they actually do. And I'm not overemphasizing the point, either. If we could beat this fundamental metabias and see what every hypothesis really predicted, we would be able to recover from almost any other error of fact.
The mirror challenge for decision theory is seeing which option a choice criterion really endorses. If your stated moral principles call for you to provide laptops to everyone, does that really endorse buying a $1 million gem-studded laptop for yourself, or spending the same money on shipping 5000 OLPCs?
We seem to have evolved a knack for arguing that practically any goal implies practically any action. A phlogiston theorist explaining why magnesium gains weight when burned has nothing on an Inquisitor explaining why God's infinite love for all His children requires burning some of them at the stake.
There's no mystery about this. Politics was a feature of the ancestral environment. We are descended from those who argued most persuasively that the good of the tribe meant executing their hated rival Uglak. (We sure ain't descended from Uglak.)
And yet... is it possible to prove that if Robert Mugabe cared only for the good of Zimbabwe, he would resign from its presidency? You can argue that the policy follows from the goal, but haven't we just seen that humans can match up any goal to any policy? How do you know that you're right and Mugabe is wrong? (There are a number of reasons this is a good guess, but bear with me here.)
Human motives are manifold and obscure, our decision processes as vastly complicated as our brains. And the world itself is vastly complicated, on every choice of real-world policy. Can we even prove that human beings are rationalizing—that we're systematically distorting the link from principles to policy—when we lack a single firm place on which to stand? When there's no way to find out exactly what even a single optimization criterion implies? (Actually, you can just observe that people disagree about office politics in ways that strangely correlate to their own interests, while simultaneously denying that any such interests are at work. But again, bear with me here.)
Where is the standardized, open-source, generally intelligent, consequentialist optimization process into which we can feed a complete morality as an XML file, to find out what that morality really recommends when applied to our world? Is there even a single real-world case where we can know exactly what a choice criterion recommends? Where is the pure moral reasoner—of known utility function, purged of all other stray desires that might distort its optimization—whose trustworthy output we can contrast to human rationalizations of the same utility function?
Why, it's our old friend the alien god, of course! Natural selection is guaranteed free of all mercy, all love, all compassion, all aesthetic sensibilities, all political factionalism, all ideological allegiances, all academic ambitions, all libertarianism, all socialism, all Blue and all Green. Natural selection doesn't maximize its criterion of inclusive genetic fitness—it's not that smart. But when you look at the output of natural selection, you are guaranteed to be looking at an output that was optimized only for inclusive genetic fitness, and not the interests of the US agricultural industry.
In the case histories of evolutionary science—in, for example, The Tragedy of Group Selectionism—we can directly compare human rationalizations to the result of pure optimization for a known criterion. What did Wynne-Edwards think would be the result of group selection for small subpopulation sizes? Voluntary individual restraint in breeding, and enough food for everyone. What was the actual laboratory result? Cannibalism.
Now you might ask: Are these case histories of evolutionary science really relevant to human morality, which doesn't give two figs for inclusive genetic fitness when it gets in the way of love, compassion, aesthetics, healing, freedom, fairness, et cetera? Human societies didn't even have a concept of "inclusive genetic fitness" until the 20th century.
But I ask in return: If we can't see clearly the result of a single monotone optimization criterion—if we can't even train ourselves to hear a single pure note—then how will we listen to an orchestra? How will we see that "Always be selfish" or "Always obey the government" are poor guiding principles for human beings to adopt—if we think that even optimizing genes for inclusive fitness will yield organisms which sacrifice reproductive opportunities in the name of social resource conservation?
To train ourselves to see clearly, we need simple practice cases.
So what are the forces at play in this scenario? Evolution will tend toward optimizing the replication of anything that replicates; the environment consists of inanimate objects and other entities that evolution is tending to optimize; each entity is optimized at an individual level despite interactions with other replicating entities. There is no foresight. Evolution will show no favoritism among the various replicating entities.
Gene fragments, genes, organelles, cells, individuals, family, and society are all replicating entities, and all can be selected for -- though selection for each of the components occurs first and more often, and entirely uninterested in higher levels of selection. Information encoded in neural networks is also a replicating entity.
Consider for example a human: ~3.2 billion base pairs of DNA, comprising ~25,000 genes, on 23 chromosomes, and all the previous mostly doubled to make a diploid cell. Mendelian reproduction serves to enforce some cooperation of the various genes. Early differentiation of cells into reproductive and somatic cells serves to enforce cooperation at the cell level; somatic cells won't reproduce indefinitely, but can assist the reproduction of the gamete cells. These mechanisms work pretty well, though despite their severity there are exceptions -- for example meiotic drive and retrotransposons allow genes to cheat Mendelian reproduction, and transmissible cancer as seen devastating the Tasmanian devils shows cells can successfully go rouge. Social enforcement mechanisms exist, but are mild compared to the aforementioned methods.
Humans also contain information stored in the brain, which can be modified and transmitted (though a proper model of that would be like creating an artificial general intelligence). Ideas are not tied to the genes, and are transmitted independently of the genes of the humans holding them -- so why shouldn't there be ideas that act in opposition to the genes of the human holding them? It would be quite the achievement for evolution to produce humans immune to ideas harmful to their genes, while still keeping the enormously useful capability to generate and transmit ideas.
As a side note, consider the search space of evolution. The request, "Find the strand of DNA size 3.2 billion base pairs in length, that is optimal for reproduction in [this environment]" consists of a search space of over 4^3,200,000,000. (And the actual search space is indefinitely larger.) Even an entity with access to the combined resources of the entire universe isn't going to be able to look through that search space.