I was in an exchange here a while back about this question. Komponisto and I basically agree that a trial, whether the guilt is decided by a judge or jury, should work like this:
For a conviction, they (jury or judge) must find that the odds of guilt meet some specific, high threshold, such as 100:1 odds.
They should be allowed to incorporate any relevant Bayesian evidence (such as "probability that a random person committed this crime")
But there should be a list of exceptions to the above for cases where cognitive biases significantly distort the evidence (i.e. their sex life, their religion, hearsay, etc.) or where the society has made a conscious decision that something shouldn't matter or be presented, irrespective of how informative it is (i.e. race/gender of the accused, illegally gathered evidence, stuff endangering national security, etc.).
Retain the adversarial system, but require each side to present (and the jury be trained in) a causal/Bayes network indicating what they think happened and how strong each piece of evidence is within that net.
The jury should be required to construct their own diagram (possibly borrowing from one side) and determine the weights of each piece of evidence with it, and then use it to compute the posterior of the defendant's guilt, which can then be compared against the guilt threshold.
This last one is important because it will clarify the presence of any poor inference on the part of the jury. While the jury can just fudge the numbers to match their "gut feeling", they cannot do so without assigning unreasonable evidence weight somewhere in the diagram, allowing the public to see if the jury wasn't thinking right. ("Whoa, you gave eight freaking bits of evidence toward guilt in a homicide case because the defendant "dresses like a slut"???)
Beyond that, where I may or may not agree with komponisto, I think the justice system should also have these properties:
Be continuous rather than binary: For at least some cases, it shouldn't be all-or-nothing, but the compensation paid to the alleged victim should vary with the probability of guilt the jury finds.
Also, punishments should be amplified by the inverse of the recovery rate (basically, the rate at which that crime is solved), with the extra compensation going to a victims' fund to pay those for whom no suspect is ever caught.
The issue of jail should be less coupled to the issue of compensation and guilt. Whether someone is jailed should be determined by how dangerous they are expected to be, as judged by some insurer, whether or not they are found guilty. Whether they should pay compensation or be punished should be determined by causal culpability, whether or not the accused is generally dangerous.
This decoupling is most important for cases like drunk driving or accidental homicide. If it's something that the accused is unlikely to do again, they shouldn't be jailed, but they should have to pay compensation anyway to those they hurt. In cases of drunk driving that don't hurt anyone, the evidence that the person drove recklessly while drunk should be fed to their liability insurer, who can then update the risk assessment of the insured.
Be continuous rather than binary: For at least some cases, it shouldn't be all-or-nothing, but the compensation paid to the alleged victim should vary with the probability of guilt the jury finds.
I basically agree here, though it should be noted that determining exactly how the compensation should vary with the probability would be a difficult problem. (You certainly wouldn't want it to be directly proportional; I would after all shudder to think of the LW "verdict" imposing a sentence of (0.35)(26) = 9.1 years on poor Amanda Knox.)
...Also, pun
The following will explore a couple of areas in which I feel that the criminal justice system of many Western countries might be deficient, from the standpoint of rationality. I am very much interested to know your thoughts on these and other questions of the law, as far as they relate to rational considerations.
Moral Luck
Moral luck refers to the phenomenon in which behaviour by an agent is adjudged differently based on factors outside the agent's control.
Suppose that Alice and Yelena, on opposite ends of town, drive home drunk from the bar, and both dazedly speed through a red light, unaware of their surroundings. Yelena gets through nonetheless, but Alice hits a young pedestrian, killing him instantly. Alice is liable to be tried for manslaughter or some similar charge; Yelena, if she is caught, will only receive the drunk driving charge and lose her license.
Raymond, a day after finding out that his ex is now in a relationship with Pardip, accosts Pardip at his home and attempts to stab him in the chest; Pardip smashes a piece of crockery over Raymond's head, knocking him unconscious. Raymond is convicted of attempted murder, receiving typically 3-5 years chez nous (in Canada). If he had succeeded, he would have received a life sentence, with parole in 10-25 years.
Why should Alice be punished by the law and demonized by the public so much more than Yelena, when their actions were identical, differing only by the sheerest accident? Why should Raymond receive a lighter sentence for being an unsuccessful murderer?
Some prima facie plausible justifications:
Trial by Jury; Trial by Judge
Those of us who like classic films may remember 12 Angry Men (1957) with Henry Fonda. This was a remarkably good film about a jury deliberating on the murder trial of a poor young man from a bad neighbourhood, accused of killing his father. It portrays the indifference (one juror wants to be out in time for the baseball game), prejudice and conformity of many of the jurors, and how this is overcome by one man of integrity who decides to insist on a thorough look through the evidence and testimony.
I do not wish to generalize from fictional examples; however, such factors are manifestly at play in real trials, in which Henry Fonda cannot necessarily be relied upon to save the day.
Komponisto has written on the Knox case, in which an Italian jury came to a very questionable (to put it mildly) conclusion based on the evidence presented to them; other examples will doubtless spring to mind (a famous one in this neck of the woods is the Stephen Truscott case - the evidence against Truscott being entirely circumstantial.
More information on trial by jury and its limitations may be found here. Recently the UK has made some moves to trial by judge for certain cases, specifically fraud cases in which jury tampering is a problem.
The justifications cited for trial by jury typically include the egalitarian nature of the practice, in which it can be guaranteed that those making final legal decisions do not form a special class over and above the ordinary citizens whose lives they effect.
A heartening example of this was mentioned in Thomas Levenson's fascinating book Newton and the Counterfeiter. Being sent to Newgate gaol was, infamously in the 17th and 18th centuries, an effective death sentence in and of itself; moreover, a surprisingly large number of crimes at this time were capital crimes (the counterfeiter whom Newton eventually convicted was hanged). In this climate of harsh punishment, juries typically only returned guilty verdicts either when evidence was extremely convincing or when the crime was especially heinous. Effectively, they counteracted the harshness of the legal system by upping the burden of proof for relatively minor crimes.
So juries sometimes provide a safeguard against abuse of justice by elites. However, is this price for democratizing justice too high, given the ease with which citizens naive about the Dark Arts may be manipulated? (Of course, judges are by no means perfect Bayesians either; however, I would expect them to be significantly less gullible.)
Are there any other systems that might be tried, besides these canonical two? What about the question of representation? Does the adversarial system, in which two sides are represented by advocates charged with defending their interests, conduce well to truth and justice, or is there a better alternative? For any alternatives you might consider: are they naive or savvy about human nature? What is the normative role of punishment, exactly?
How would the justice system look if LessWrong had to rewrite it from scratch?