On the Freakonomics blog, Steven Pinker had this to say:

There are many statistical predictors of violence that we choose not to use in our decision-making for moral and political reasons, because the ideal of fairness trumps the ideal of cost-effectiveness. A rational decision-maker using Bayes’ theorem would say, for example, that one should convict a black defendant with less evidence than one needs with a white defendant, because these days the base rates for violence among blacks is higher. Thankfully, this rational policy would be seen as a moral abomination.

I've seen a common theme on LW that is more or less "if the consequences are awful, the reasoning probably wasn't rational". Where do you think Pinker's analysis went wrong, if it did go wrong?

One possibility is that the utility function to be optimized in Pinker's example amounts to "convict the guilty and acquit the innocent", whereas we probably want to give weight to another consideration as well, such as "promote the kind of society I'd wish to live in".