I agree that the present-day U.S. prisons are absolutely horrible. On the other hand, their awfulness is an important element of deterrence, so it's not entirely clear what the net consequences of making them less awful would be.

With such high recidivism rates, though, deterrence doesn't seem to be working too well.

As I think about this further, it occurs to me that maybe the deterrent effect partially results from people's fear of the unknown. Once a person has been in prison, even with its awfulness, it might not seem as bad. (There's also the problem that many repeat offenders come from environments so unstable that prison hardly seems bad by comparison, no matter what.)

It is also unclear what would be the cost of keeping such a large prison population in much better conditions.

One problem is the unreasonable size of the population itself, due in large part to the absurdity of U.S. drug laws and penalties. The large size of the prison population serves to point out that we should be trying to improve the system in a variety of ways. Also, just to clarify, when I made the Rawlsian argument about prisons, I was referring more to the incarceration vs. rehabilitation debate than the awfulness of conditions in general.

On the whole, it seems to me that what evidence exists suggests that the harsh approach emphasizing deterrence and incapacitation has so far been vindicated much better in practice than the attempts to make rehabilitation feasible on a mass scale.

Yes, I agree, but we should note that deterrence and incapacitation are much easier to implement effectively than rehabilitation. Locking people up is something we've known how to do for millennia; we're not quite as practiced at understanding human psychology and neurology.