I wasn't good at making the analogy... the important thing is that there is also an arbiter (jury). When that arbiter does not even know rules of chess, you get a major problem. It is absolutely essential that arbiter knows rules of chess perfectly.
With regards to training the players the issue is that w/o training the outcome gets decided by mistake rate. The Kasparov is guaranteed to win versus me, if we were to do equal amount of training. He is not guaranteed to win vs me if we both played chess for the first time in our lives.
I wasn't good at making the analogy
I think I actually agree with the point you are trying to make with the analogy.
The Kasparov is guaranteed to win versus me, if we were to do equal amount of training. He is not guaranteed to win vs me if we both played chess for the first time in our lives.
I would still place the difference the other way. I'd give naive Kasparov (even) better odds against naive you than I would give trained Kasparov against a you of equal training and experience.
I'm interested in how courts and juries might use rational techniques to arrive at correct decisions on guilt.
In a complex case, it would seem to sensible to assess each component of the prosecution and defence case, and estimate the relative likelihood. If the prosecution case is (say) 100 times more likely than the defence case, then you can say the defendant is guilty beyond reasonable doubt.
I never heard of this being done though. I recently made an analysis of the Massei report into the Amanda Knox case. It looked like this ( see http://massei-report-analysis.wikispaces.com/ for the entire analysis and some insight into the numbers below ).
This is perhaps a bit vague. It's not a great example, because in the end I didn't find any credible prosecution evidence. It's not entirely clear what the "probability" numbers here actually are, and whether two columns are needed. But hopefully it shows that the Massei's account of the murder is quite improbable, and there is considerable doubt.
I'm interested in possibly devising a more complete framework for how such an assessment should be done, the pitfalls that need to be guarded against (how uncertain are the probability estimates?), and even views as to how "reasonable doubt" should be quantified.
Perhaps readers would like to make an assessment of other interesting cases, to explore the issues.
Or how would you approach this problem?