= 27d567bc2d48d9f40e9ccc20ea370b15)
Psychohistorian comments on Hearsay, Double Hearsay, and Bayesian Updates - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (105)
That's an excellent question. The answer depends on exactly what you mean by "better than chance." If you mean "more than half of those convicted of a crime are guilty of that crime," then I'd say yes, there is excellent reason to think that they are. Prosecutors usually have access to several times more reports of crime than they can afford to go out and prosecute. Prosecutors are often explicitly or implicitly evaluated on their win ratio -- they have strong incentives to pick the 'easy' cases where there is abundant evidence that the suspect is guilty. Most defense lawyers will cheerfully concede that the vast majority of their clients are guilty -- either the clients admit as much to their lawyers, or the clients insist on implausible stories that don't pass muster, which the lawyers have to disguise in order to get their clients to go free. Although as a matter of law and rhetoric people are presumed innocent until proven guilty, as a matter of cold statistics, someone who has been lawfully indicted in America is probably more likely to be guilty than innocent. In fact, there are probably so many guilty suspects in Court that the legal system does strictly worse than what social scientists call a "naive predictor" -- i.e., just assuming that everyone is guilty. Of course, that wouldn't be a sustainable policy -- prosecutors choose easy cases because they know that they'll be required to win those cases in a relatively challenging environment. If the rule were that everyone is guilty, prosecutors would start choosing cases based on other criteria, and the percentage of indicted suspects who were actually guilty would go down.
Suppose you survey defense attorneys, and conclude that, say, roughly 80% of indicted suspects are guilty. Could you somehow measure whether the legal system does better than a "mixed strategy predictor" that guessed that a suspect was guilty with probability 0.8 and guessed that a suspect was innocent with probability 0.2? The mixed-strategy predictor would get an accurate result in (0.8) ^ 2 + (0.2) ^ 2 = 68% of the time. To assess whether the legal system is better than a mixed-strategy predictor, you would need to have a way of validating at least a sample of actual cases. I really have no idea how you would start to do that. It's not clear that self-reported guilt or defense-attorney-assessed guilt will correlate strongly enough with actual guilt that we can figure out which individual cases the legal system gets right and which ones it gets wrong. But if you can't measure accuracy in individual cases, how do you figure out the system's overall accuracy rate? It's not clear that looking at appellate results or DNA exonerations, etc. would help either. A reversal on appeal is no guarantee of innocence, because a sentence can be reversed (a) if the evidence is still strong but not strong enough to remove all reasonable doubt as well as (b) when the prosecution or police have used inappropriate but reliable tactics (such as using high-tech cameras to take pictures of the inside of your home without a warrant).
Finally, there is "better than chance" in the sense of specific forensic techniques being verifiably better than, say, a Ouija board. There are several pretty good techniques, such as document analysis, DNA analysis, electronic tracing, and perhaps even paired-question polygraph testing. Whether or not the system interprets the evidence correctly, a typical trial at least contains sufficient evidence for a rational evaluator to beat chance.
A few observations:
You confuse different parts of the justice system, and your criticism is internally self-contradictory. If everyone who ever lived is guilty of something, then high or racially disparate incarceration rates need not catch innocent people. The fact that this occurs is more an indictment of the laws in place and the people who prosecute them, not the courts that adjudicate them.
Put another way, if breathing were a crime, then everyone convicted of it would in fact be guilty. If there were a lot of black men convicted of it, more so than other races, it would likely be due to different rates of prosecution, given how easy the charge would be to prove this is bad, but it is a criticism of the wrong part of the government. It would be like blaming the mayor for the ineffectiveness of the postal service (the former is the city government, the latter is federal).
Edited to clarify: I am referring to the value of our adversarial method and our rules of evidence / constitutional protections - the mechanics of how a trial works. There is an entirely separate issue of prosecutorial discretion and unequal police enforcement and overly draconian laws, which certainly lead to the problems being discussed here. But the entire purpose of evidence law and courtroom proceedings generally is to determine if the person charged is in fact guilty. It is not to determine if the prosecution is charging the right people or if the laws are just or justly enforced. So this criticism seems misplaced.
Selective prosecution is a problem that can be laid at the prosecutor's doorstep, and if proven, it's a basis for acquittal. The problem is that it's hard to prove, not that the prosecutor and cops aren't responsible. (See my essay "Threat to advocacy from overdeterrence."
If a bad law is applied in a racist way, surely that's a problem with both the law itself and the justice system's enforcement of it?
There's nothing self-contradictory about my criticism, there are multiple interrelated problems. American legal system is totally broken, it's not just one easy fix away from sanity.
Fetishization of the concept of being "guilty" is one of these problems.
For example - everybody on less wrong is obviously either "guilty" of conspiracy to commit treason and replace various governments by Friendly AI, or at least "guilty" of helping people directly involved in such conspiracy. That's a problem.
If someone rounded up whoever on lesswrong happens to be black, and prosecuted them, that's would be a second problem.
It is self contradictory on its face . Compare these statements:
The first statement provides a complete alternative explanation for the second two. It is entirely possible to believe that (A) there are far too many crimes, (B) police and prosecutors are biased against black people, and this fully explains why there are so many black people in prison without a single one of them needing to be innocent. Similarly, you say that, given crime rates and incarceration rates, some people must be innocent. Again, this is undermined by the fact that you say everyone is guilty of something. You just can't argue that everyone is a criminal, and then argue that high incarceration rates must necessarily be attributable to a high rate of convicting innocent persons. They may both be true, but you have no basis to infer the latter given the former.
To the extent that you disclaim this by saying that people are "guilty" of minor "crimes," your argument becomes largely circular, and is still not supported by evidence. What percentage of thieves/murderers/rapists/etc. are actually caught? How long are sentences? Combine a higher crime rate with a higher catch rate and longer sentences and you easily get a huge prison population without innocent people being convicted. I don't claim to know if this is the case, but you do, so you need to back it up.
There are good reasons to believe few trials that happen are extremely far from any kind of fairness, and they're stacked to give persecution an advantage. Just compare massive funding of police and prosecutors with puny funding of defense attorneys.
You completely confuse "did something for which an unfriendly legal system can find them guilty" and "did something seriously dangerous to others that could sensibly be a reason to lock someone one".
These are not even remotely close. Essentially everyone is the former, very few people are the latter.
If your point is that there are a lot of people locked up for violating laws that are basically stupid, you're absolutely right.
But that issue is largely irrelevant to the subject of the primary post, which is the accuracy of courts. If the government bans pot, the purpose of evidence law is to determine whether people are guilty of that crime with accuracy.
In other words, your criticism of the normative value of the American legal system is spot-on; we imprison far more people than we should and we have a lot of stupid statutes. But since this context is a discussion of the accuracy of evidentiary rules and court procedure, your criticism is off-topic.
Is it really off-topic to suggest that looking at the accuracy of the courts may amount to rearranging the deck chairs on the titanic in a context where we've basically all agreed that
the courts are not terrible at making accurate determinations of whether a defendant broke a law
The set of laws where penalties can land you in prison are massively inefficient socially and in most people's minds unjust (when we actually grapple with what the laws are, as opposed to how they are usually applied to people like us, for those of us who are white and not poor).
The system of who is tried versus who makes plea bargains versus who never gets tried is systematically discriminatory against those with little money or middle/upper class social connections, and provides few effective protections against known widespread racial bias on the part of police, prosecutors and judges.
How different is this in principle from TimS's suggestion about lower hanging fruit within evidentiary procedure, just at a meta level? Or did you consider that off-topic as well?
Can you cite evidence for this?
Most of the evidence for this is based on arguing that P(conviction|African descent) > P(conviction|Eurasian descent) and dismissing anyone who points out that P(guilty|African descent) > P(guilty|Eurasian descent) as a racist.
(Or even do the homophone and 'cite'?)
That is Bayesian evidence that there is a racial bias in the conviction process.
(But that isn't evidence!)
Mind you those people are probably being racist. In particular they are 'pointing out' rather than, say, hypothesizing.
P(guilty|African decent) > P(guilty|Eurasian decent) alright, but that kind of evidence is easily screened out. For any non-trivial amount of forensic evidence E, P(guilty|E, African descent) ought to be approximately the same as P(guilty|E, Eurasian descent). (For example, if E points toward the defendant being guilty, even though you'd assign a higher prior probability of guilt for a black person than for a white person, you'd assign a higher prior probability of E for a black person than for a white person too, so that in the posterior probabilities those cancel out.)
The original question was:
which I would interpret as ‘Is P(Conviction|Guilt) substantially larger than P(Conviction|Innocence)?’ Now, for some crimes such as copyright infringement, P(G) is very close to 1, so P(C|G) cannot be close to 1 simply because then there wouldn't be enough room in prisons to hold NP(C|G)P(G) people (N being the population -- times the mean sentence length, over the mean lifespan, and possibly some other factor of order unity I'm forgetting of), and since P(I) is small, in order for P(C|I) to be much less than P(C|G), P(C and I) = P(C|I)P(I) must be very small.
(Also, we want the system to be unbiased, i.e. P(C|G, brown skin) to be close to P(C|G, pink skin), P(C|G, penis) to be close to P(C|G, vagina), and so on, and so forth. The best way of achieving this would IMO be for all of these numbers to be close to 1, but that's impossible with the current definition of G and finite capacity of prisons.)
Are we restricting to cases that are prosecuted or doing this over all people?
The immediately obvious answer would be “over the prosecuted if you're only ‘testing’ the courtroom itself, over all the people if you're ‘testing’ the whole system”, but I'm not sure what the ‘ideal’ thing for a courtroom to do in terms of P(C|Prosecution, G) and P(C|P,I) if the police is ‘non-ideal’ so that P(P|G) is not close to or greater than P(P|I) to start with. Or even whether this question makes sense... I'll have to think more clearly about this when I'm not this tired.
It should be noted that the reasonable meaning of 'substantially' in such a case is relative not just the unadorned absolute difference in the ridiculously low probabilities. In comparisons of this nature a difference it is overwhelmingly obvious that "0.0001 - 0.000001" is much more significant than "0.6001 - 0.600001".
The above consideration is even more important when the substitution of "is significantly better than chance" with "‘Is P(Conviction|Guilt) substantially larger than P(Conviction|Innocence)?’" is yours and not the original question.
I'm not sure of this: I'd very much prefer a world where 0.01% of guilty people and 0.0001 of innocent people are convicted to one where 60.01% and 60.001% are. (If convicting a guilty person has utility A and convicting an innocent person has utility -B, what you want to maximise is AP(C|G) - BP(C|I), which also depends on the magnitudes of the probabilities, not only on their ratio.)
Huh? How else could the original question be interpreted? [ETA: Well, if better is interpreted instrumentally rather than epistemically, “the legal system is significantly better than chance” means “AP(C|G) - BP(C|I) is significantly greater than AP(C) - BP(C)”; and since A is orders of magnitude less than B (unless we're talking about serial killers or similarly serious stuff), that boils down to “P(C|I) is significantly less than P(C)”, where it's the magnitude of the difference that matters, rather than the ratio.]