There's an argument (first advanced by Beccaria in the late 18th century) that it matters more that punishment be swift and certain, than that it be harsh. If people don't really believe a punishment is likely to happen to them, it won't deter reliably. Human cognitive biases being what they are, we might be better served trying to make punishment visible, rather than horrifying. Azkaban, being remote and unpleasant to think about, is perhaps less effective than some punishment that would be constantly in sight. Having the convicted criminal's wand broken. say.

Beccaria puts it much better than I could, so I'll just refer you to his essay on the topic: http://en.wikisource.org/wiki/An_Essay_on_Crimes_and_Punishments/Chapter_XXVII

In a society with veritaserum, legilimency and assorted other magic you'd think it would be straightforward to establish guilt or innocence in the vast majority of cases.