Continuing my interest in tracking real-world predictions, I notice that the recent acquittal of Knox & Sollecito offers an interesting opportunity - specifically, many LessWrongers gave probabilities for guilt back in 2009 in komponisto’s 2 articles:
- “You Be the Jury: Survey on a Current Event”
- “The Amanda Knox Test: How an Hour on the Internet Beats a Year in the Courtroom”
Both were interesting exercises, and it’s time to do a followup. Specifically, there are at least 3 new pieces of evidence to consider:
- the failure of any damning or especially relevant evidence to surface in the ~2 years since (see also: the hope function)
- the independent experts’ report on the DNA evidence
- the freeing of Knox & Sollecito, and continued imprisonment of Rudy Guede (with reduced sentence)
Point 2 particularly struck me (the press attributes much of the acquittal to the expert report, an acquittal I had not expected to succeed), but other people may find the other 2 points or unmentioned news more weighty.
2 Probabilities
I was curious how the consensus has changed, and so, in some spare time, I summoned all the Conscientiousness I could and compiled the following list of 54 entries based on those 2 articles’ comments (sometimes inferring specific probabilities and possibly missing probabilities given in hidden subthreads), where people listed probabilities for Knox’s guilt, Sollecito’s guilt, and Guede’s guilt:
Knox | Sollecito | Guede | LWer |
---|---|---|---|
.20 | .20 | .70 | badger |
.05 | .10 | .90 | mattnewport |
.20 | .25 | .90 | AngryParsley |
.05 | .05 | .95 | tut |
.05 | .05 | .95 | bentarm |
.85 | .60 | .20 | bgrah449 |
.01 | .01 | .99 | kodos96 |
.01 | .01 | .99 | Daniel_Burfoot |
.40 | .40 | .90 | nerzhin |
.45 | .45 | .60 | Matt_Simpson |
.33 | .33 | .90 | Cyan |
.50 | .50 | .95 | jimmy |
.05 | .05 | .99 | Psychohistorian |
.40 | .40 | .90 | Threads |
.50 | .50 | .80 | Morendil |
.15 | — | — | Eliezer_Yudkowsky |
.20 | .35 | .98 | LauraABJ |
.10 | .10 | .90 | curious |
.20 | .20 | .96 | jpet |
.06 | .06 | .70 | saliency |
.80 | .60 | .95 | Mario |
.20 | .20 | .95 | Yvain |
.70 | — | — | Shalmanese |
.05 | .05 | .95 | gelisam |
.05 | .05 | .90 | Mononofu |
.90 | .90 | .90 | lordweiner27 (changed mind) |
.50 | .50 | .99 | GreenRoot |
.99 | .99 | .99 | dilaudid |
.13 | .15 | .97 | Jack |
.05 | .05 | .90 | wedrifid |
.01 | .01 | .90 | Nanani |
.35 | .35 | .95 | imaxwell |
.01 | .01 | .99 | jenmarie |
.25 | .25 | .75 | Jawaka |
.41 | .38 | .99 | magfrump |
.40 | .20 | .60 | gwern |
.08 | .10 | .95 | loqi |
.25 | .25 | .50 | JamesAndrix |
.90 | .85 | .99 | Unknowns |
.35 | .35 | .90 | Sebastian_Hagen |
.90 | .90 | .99 | brazil84 |
.30 | .30 | .40 | ChrisHibbert |
.02 | .02 | .98 | wnoise |
.50 | .40 | .90 | John_Maxwell_IV |
.10 | .10 | — | k3nt |
.01 | .01 | .99 | Sinai |
.00 | .00 | 1.0 | KayPea |
.00 | .00 | .60 | MerleRideout |
.15 | .10 | .80 | TheRev |
.01 | .01 | .99 | komponisto |
.30 | — | — | pete22 |
.01 | — | — | SforSingularity |
.00 | .00 | .90 | AnnaGilmour |
.05 | .05 | .95 | Seth_Goldin |
.60 | .60 | .95 | bigjeff5 |
It’s interesting how many people assign a high-probability to Knox being guilty; I had remembered LW as being a hive of Amanda fans, but either I’m succumbing to hindsight bias or people updated significantly after those articles. (For example, Eliezer says .15 is too high, but doesn’t seem otherwise especially convinced; and later one reads in Methods of Rationality that "[Hagrid] is the most blatantly innocent bystander to be convicted by the magical British legal system since Grindelwald's Confunding of Neville Chamberlain was pinned on Amanda Knox.")
EDIT: Jack graphed the probability against karma:
2.1 Outliers
If we look just at >41% (chosen to keep contacts manageable), we find 12 entries out of 54:
Knox | Sollecito | Guede | LWer |
---|---|---|---|
.45 | .45 | .60 | Matt_Simpson |
.50 | .40 | .90 | John_Maxwell_IV |
.50 | .50 | .80 | Morendil |
.50 | .50 | .95 | jimmy |
.50 | .50 | .99 | GreenRoot |
.60 | .60 | .95 | bigjeff5 |
.70 | — | — | Shalmanese |
.80 | .60 | .95 | Mario |
.85 | .60 | .20 | bgrah449 |
.90 | .85 | .99 | Unknowns |
.90 | .90 | .90 | lordweiner27 |
.90 | .90 | .99 | brazil84 |
.99 | .99 | .99 | dilaudid |
I have messaged each of them, asking them to comment here, describing if and how they have since updated, and any other thoughts they might have. (I have also messaged the first 12 commenters or so, chronologically, with <41% confidence in Knox’s guilt, with the same message.) The commenters:
AngryParsley / Cyan / Daniel_Burfoot / Eliezer_Yudkowsky / GreenRoot / John_Maxwell_IV / LauraABJ / Mario / Matt_Simpson / Morendil / Psychohistorian / Shalmanese / Threads / Unknowns / badger / bentarm / bgrah449 / bigjeff5 / brazil84 / dilaudid / jimmy / kodos96 / lordweiner27 / mattnewport / nerzhin / tut
I look forward to seeing their retrospectives, or indeed, anyone's retrospectives on the matter.
- Allknowing and most merciful Bayes;
- We have erred, and strayed from thy ways like biased sheep.
- We have followed too much the devices and desires of our own hearts.
- We have offended against thy axiomatic laws.
- We have left undone those updates which we ought to have done;
- And we have done those updates which we ought not to have done;
- And there is no calibration in us.
- But thou, O Bayes, have mercy upon us, miserable wannabes.
- Spare thou them, O Bayes, who confess their faults.
The Knox thread was one of the first steps in my getting interested in predictions in general. It was a slow process and is still ongoing, but it has had me spend time on various calibration exercises, on PredictionBook, on the Crowdcast instance dedicated to the Good Judgment project, on Inkling Markets because I saw a few arbitrage opportunities there that sounded like fun. I'm not as into predictions as gwern appears to be, but they're growing on me.
All that and I'm still not very sure what to think of the Knox case. Yes, if our predictions were being scored I'd be getting a non-trivial penalty from my 50% chance of her guilt - that is, if we take the outcome of the appeals process as an arbitration of the prediction, and judge, for the purposes of scoring, that she was "in fact" innocent. (I'm not saying I have much doubt now about her innocence: I'm saying that we won't ever know for sure, and part of the point of these prediction exercises is to allow us to better deal with that permanent uncertainty.)
On the other hand, some of the people listed above would be taking a much more serious hit. One thing I've learned from my various exercises is that you can't expect to be right all the time - sometimes, with minimal knowledge of the relevant facts, a 50% prediction is in fact not so bad.
Then again, some of us were also apparently very confident in the answer that "in fact" turned out to be the correct one. Then again, we all get lucky from time to time - that too is the nature of the beast...
My intention is to continue to learn, to continue to get better at predicting, to become better calibrated and more discriminating over time.
One of the unfortunate things about living when we do is that it seems unlikely there will be any future oracles developed which reveal definitive answers to ancient crimes.
I refer, of course, to DNA evidence, which gave us an astonishing oracle to ask questions about old crimes, revealing a shockingly high lower bound on the justice syst... (read more)