With such high recidivism rates, though, deterrence doesn't seem to be working too well.

Not necessarily. Even very effective deterrence that drastically cuts crime may coexist with high recidivism rates, since those who ever get imprisoned in the first place are those for whom deterrence is less effective. A high percentage of repeat offenders among former prisoners doesn't mean that there isn't a huge population of potential offenders that are successfully deterred altogether. (Whether this is true is of course arguable.)

One problem is the unreasonable size of the population itself, due in large part to the absurdity of U.S. drug laws and penalties.

I don't think "absurdity" is the right word. Drug laws are among those selectively enforced sweeping provisions that give the authorities a broad enough latitude to use them as pretext for enforcement of unwritten de facto norms which would be nowadays found unconstitutional or at least perceived as scandalous if spelled out explicitly, even though all respectable people hypocritically expect them to be enforced for their own benefit. Here I especially have in mind those norms that used to be enforced more openly through anti-vagrancy, anti-loitering, and similar laws. These are no longer considered constitutional, but respectable people still want the authorities to keep the underclass away from them.

In a sense, drug laws are for the underclass what the insider trading, arcane tax violations, conspiracy, obstruction of justice, and similar vague and sweeping laws are for the rich.

As I think about this further, it occurs to me that maybe the deterrent effect partially results from people's fear of the unknown. Once a person has been in prison, even with its awfulness, it might not seem as bad.

Also, a significant element of any criminal punishment is the subsequent stigma of having been subjected to it, both informal and official (as reflected in background checks). This factor is clearly less relevant for repeated imprisonment.