Douglas_Knight comments on Hearsay, Double Hearsay, and Bayesian Updates - Less Wrong

47 Post author: Mass_Driver 16 February 2012 10:31PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (105)

You are viewing a single comment's thread. Show more comments above.

Comment author: Mass_Driver 16 February 2012 12:18:04AM 25 points [-]

That's an excellent question. The answer depends on exactly what you mean by "better than chance." If you mean "more than half of those convicted of a crime are guilty of that crime," then I'd say yes, there is excellent reason to think that they are. Prosecutors usually have access to several times more reports of crime than they can afford to go out and prosecute. Prosecutors are often explicitly or implicitly evaluated on their win ratio -- they have strong incentives to pick the 'easy' cases where there is abundant evidence that the suspect is guilty. Most defense lawyers will cheerfully concede that the vast majority of their clients are guilty -- either the clients admit as much to their lawyers, or the clients insist on implausible stories that don't pass muster, which the lawyers have to disguise in order to get their clients to go free. Although as a matter of law and rhetoric people are presumed innocent until proven guilty, as a matter of cold statistics, someone who has been lawfully indicted in America is probably more likely to be guilty than innocent. In fact, there are probably so many guilty suspects in Court that the legal system does strictly worse than what social scientists call a "naive predictor" -- i.e., just assuming that everyone is guilty. Of course, that wouldn't be a sustainable policy -- prosecutors choose easy cases because they know that they'll be required to win those cases in a relatively challenging environment. If the rule were that everyone is guilty, prosecutors would start choosing cases based on other criteria, and the percentage of indicted suspects who were actually guilty would go down.

Suppose you survey defense attorneys, and conclude that, say, roughly 80% of indicted suspects are guilty. Could you somehow measure whether the legal system does better than a "mixed strategy predictor" that guessed that a suspect was guilty with probability 0.8 and guessed that a suspect was innocent with probability 0.2? The mixed-strategy predictor would get an accurate result in (0.8) ^ 2 + (0.2) ^ 2 = 68% of the time. To assess whether the legal system is better than a mixed-strategy predictor, you would need to have a way of validating at least a sample of actual cases. I really have no idea how you would start to do that. It's not clear that self-reported guilt or defense-attorney-assessed guilt will correlate strongly enough with actual guilt that we can figure out which individual cases the legal system gets right and which ones it gets wrong. But if you can't measure accuracy in individual cases, how do you figure out the system's overall accuracy rate? It's not clear that looking at appellate results or DNA exonerations, etc. would help either. A reversal on appeal is no guarantee of innocence, because a sentence can be reversed (a) if the evidence is still strong but not strong enough to remove all reasonable doubt as well as (b) when the prosecution or police have used inappropriate but reliable tactics (such as using high-tech cameras to take pictures of the inside of your home without a warrant).

Finally, there is "better than chance" in the sense of specific forensic techniques being verifiably better than, say, a Ouija board. There are several pretty good techniques, such as document analysis, DNA analysis, electronic tracing, and perhaps even paired-question polygraph testing. Whether or not the system interprets the evidence correctly, a typical trial at least contains sufficient evidence for a rational evaluator to beat chance.

Comment author: Douglas_Knight 16 February 2012 06:13:07AM 2 points [-]

It's not clear that looking at appellate results or DNA exonerations, etc. would help either.

I'm sorry, but I can't produce any response but bewilderment. You think DNA exonerations don't give evidence about the accuracy of the system? Really?

Comment author: skepsci 16 February 2012 07:20:33AM *  7 points [-]

It proves that mistakes have been made, but in the end, no, I don't think it's terribly useful evidence for evaluating the rate of wrongful convictions. Why not? There have been 289 post-conviction DNA exonerations in US history, mostly in the last 15 years. That gives a rate of under 20 per year. Suppose 10,000 people a year are incarcerated for the types of crime that DNA exoneration is most likely to be possible for, namely murder and rape (I couldn't find exact figures, but I suspect the real number is at least this big). Then considering DNA exonerations gives us a lower bound of something like .2% on the error rate of US courts.

That is only useful evidence about the error rate if your prior estimate of the inaccuracy was less than that, and I mean, come on, really? Only one conviction in 500 is a mistake?

Comment author: taw 16 February 2012 09:52:17AM 9 points [-]

DNA exoneration happens when one is innocent and combination of extremely lucky circumstances make retesting of evidence possible. The latter I would be shocked to find at higher than 1:100 chance.

Comment author: bigjeff5 22 February 2012 12:07:26AM -1 points [-]

The appellate system itself - of which cases involving new DNA evidence are a tiny fraction - is a much more useful measure.

There are a whole lot more exonerations via the appeals process than those driven by DNA evidence alone. This aught to be obvious, and the 0.2% provided by DNA is an extreme lower bound, not the actual rate of error correction.

Case in point, I found an article describing a study on overturning death penalty convictions, and they found that 7% of convictions were overturned on re-trial, and 75% of sentences were reduced from the death penalty upon re-trial.

One in fourteen sounds a lot more reasonable to me, and again that's just death penalty cases, for which you'd expect a higher than normal standard for conviction and sentencing.

The standard estimate is about 10% for the system as a whole.