With such high recidivism rates, though, deterrence doesn't seem to be working too well.
Not necessarily. Even very effective deterrence that drastically cuts crime may coexist with high recidivism rates, since those who ever get imprisoned in the first place are those for whom deterrence is less effective. A high percentage of repeat offenders among former prisoners doesn't mean that there isn't a huge population of potential offenders that are successfully deterred altogether. (Whether this is true is of course arguable.)
One problem is the unreasonable size of the population itself, due in large part to the absurdity of U.S. drug laws and penalties.
I don't think "absurdity" is the right word. Drug laws are among those selectively enforced sweeping provisions that give the authorities a broad enough latitude to use them as pretext for enforcement of unwritten de facto norms which would be nowadays found unconstitutional or at least perceived as scandalous if spelled out explicitly, even though all respectable people hypocritically expect them to be enforced for their own benefit. Here I especially have in mind those norms that used to be enforced more openly through anti-vagrancy, anti-loitering, and similar laws. These are no longer considered constitutional, but respectable people still want the authorities to keep the underclass away from them.
In a sense, drug laws are for the underclass what the insider trading, arcane tax violations, conspiracy, obstruction of justice, and similar vague and sweeping laws are for the rich.
As I think about this further, it occurs to me that maybe the deterrent effect partially results from people's fear of the unknown. Once a person has been in prison, even with its awfulness, it might not seem as bad.
Also, a significant element of any criminal punishment is the subsequent stigma of having been subjected to it, both informal and official (as reflected in background checks). This factor is clearly less relevant for repeated imprisonment.
All good points.
Why do those words go together?
Society - and for once, I'm using this term universally - teaches that, if you committed a crime, you should be punished.
But in some societies, we have an insanity defense. If you had a brain condition so that you had no - here it's a little vague - consciousness, or moral sense, or free will, or, well, something - then it would be cruel to punish you for your crime. Instead of going to prison, you should be placed somewhere where you can't hurt anybody, where professional physicians and counselors can study your case and try to reform you so that you can rejoin society.
Wait - so that isn't what prison is for?
No. Prison is to punish people. Is it any wonder that prison doesn't reform people, when we don't want it to reform them? Most people would be upset if prisoners could go in on Friday, and emerge, rehabilitated, on Sunday. When people say, "It would be cruel to punish people who aren't responsible for their actions", they are implicitly saying, "Prison is necessarily cruel; and that's good, because we should be cruel to criminals who are responsible for their actions."
But the more we learn about psychology and neuroscience, the further responsibility recedes into the distance.
Outcome-based justice argues that we should give up playing the blame game, because neuroscience keeps finding more and more proofs that things are "not our fault". Instead, we should write laws that deter crime.
You might think this is what we already try to do. But it isn't! Witness this confused article from the Brookings Institute, Cognitive Neuroscience and the Future of Punishment by O. Carter Snead. Snead objects to outcome-based justice. He summarized all of the arguments for it, yet managed to completely miss their point, concluding where he started from, saying that outcome-based justice is obviously bad because it could lead to being cruel to people who didn't deserve it. (Instead of only being cruel to the people who do deserve it, which is obviously what we want to do.)
Snead understands that outcome-based justice deters crime:
You might expect that Snead goes on to explain why these laws are bad things. But he doesn't! He assumes we can all see that these are obviously bad things.
The Wrath of Kahneman describes a study which asked whether people punish others in order to deter crime, and concluded, No. People are doing something else.
One theory is that people are trying to be fair. Everyone should get the same chances; everyone should get the same punishment for the same crime. John Rawls argues this explicitly in Justice as Fairness: Justice should not be utilitarian, but should instead be fair.
I believe Rawls' view is also the popular view of what "justice" means. And, I will argue in a later post, it is part of a pattern showing a deep divide between two different ways of using the word "ethics".
ADDED: Constant made the point that, while one part of outcome-based justice is preventing future harm from the criminal on the dock, another part is deterring harm by other criminals. This latter part does not benefit from punishing criminals who cannot be deterred. Thus, to optimally punish both criminals who can and cannot be deterred, the law requires a concept of moral culpability, and should punish criminals who can't be deterred more lightly. This is a better origin story for the linking together of morality and free will than the just-so story I'd come up with, so I plan on stealing it for my next post. (SilasBarta may have been trying to make the same point, but I found his comments impenetrable.)
(This post is laying groundwork for two other posts that will go in different directions, neither of which concerns justice.)