Over the summer, Eliezer suggested (approximately, I am repeating this from memory) the following method for making an important decision:
This was essentially the method I used in coming to my (probably slightly low) estimate of the probability that Knox and Sollecito were innocent. It just felt like they were innocent, and I saw essentially no reason to suspect they were guilty. I will note that the 'pro-guilt' site that komponisto linked to was just horribly devoid of anything that I might consider evidence (if anything, that site did more to convince me of Knox's innocence than the pro-innocence site), and I did spend probably about 10 minute trying to find some evidence that they had missed, but completely failed.
On a different not, as I said at the time, 0.95 and 0.05 were just proxies for "pretty damn sure" and "pretty damned unlikely" - I have very little idea what 5% probability fee...
This is a better summary of what I said than what I actually said, so I hereby declare your distorted version to be my true teaching.
Despite the fact that my opinion on the case has hardly changed at all, these posts -- and thinking about the case in general -- were a tremendous learning experience for me. Some of the lessons include:
Less Wrong is good at getting the right answer. Believe it or not, the strong survey consensus in favor of innocence -- prior to my second post -- came as something of a pleasant surprise to me. You don't find this in many other places, despite the fact that the case is a no-brainer. I had assumed there would be more wishy-washiness and probabilities close to 50% than there turned out to be. (There was some of this, but less than I expected.)
People in general are bad at getting the right answer. As shown by the original verdict, not to mention all the numerous pro-guilt commentators on internet forums and elsewhere. What's surprising about that? Not much, perhaps, but I would say that one thing that is important about it is that it shows that huge, glaring errors of judgement are not restricted to Far Mode. Even on a mundane question such as this, people are susceptible to strange cognitive biases that can severely distort their assessment of evidence.
Confidence should depend
See my added comment. I did not assign a probability of 15%. I said that if you assigned a probability higher than 15%, it meant you had a really major problem with crediting the opinions of other people and the authority of idiots. My probability that Knox and Sollecito were guilty was "that's privileging the hypothesis", i.e., "I see no real evidence in its favor so same as prior probability", i.e., "really damned' low".
That's a neat compact algorithm but this doesn't change the fact that it produces the wrong answer.
Again, 15% isn't the maximum of a range. It's a number that's not just "wrong" but "sufficiently wrong to imply you need to adjust your emotional makeup".
If you need a number for me, put in "<0.01". I wouldn't have bet $20,000 at 99-to-1 odds over it at the time of writing that first paragraph, but I'm not quite sure anymore that this really means my probability is >0.01, it's not like I'd have taken the bet the other way.
The Knox thread was one of the first steps in my getting interested in predictions in general. It was a slow process and is still ongoing, but it has had me spend time on various calibration exercises, on PredictionBook, on the Crowdcast instance dedicated to the Good Judgment project, on Inkling Markets because I saw a few arbitrage opportunities there that sounded like fun. I'm not as into predictions as gwern appears to be, but they're growing on me.
All that and I'm still not very sure what to think of the Knox case. Yes, if our predictions were being scored I'd be getting a non-trivial penalty from my 50% chance of her guilt - that is, if we take the outcome of the appeals process as an arbitration of the prediction, and judge, for the purposes of scoring, that she was "in fact" innocent. (I'm not saying I have much doubt now about her innocence: I'm saying that we won't ever know for sure, and part of the point of these prediction exercises is to allow us to better deal with that permanent uncertainty.)
On the other hand, some of the people listed above would be taking a much more serious hit. One thing I've learned from my various exercises is that you can't expect t...
Brain scans which can retrieve memories accurately?
I believe that the current understanding is that memory encoding, not just retrieval, is pretty unreliable, so even if you can read exactly what's in people's brains, it may not be much help.
My high school psychology teacher gave my class an interesting demonstration on the usefulness of eyewitnesses.
For one class, we found a notice on the door saying that the class had been relocated to another room. We went to that room, and shortly after our teacher arrived, followed by another teacher. He complained that she had not gotten proper clearance to move her class to that room, and he needed it for an exercise for his own class. They spent a couple minutes arguing, and harsh words were exchanged, after which he left the room.
Almost immediately after he left, our teacher asked us to create profiles of his physical description. Estimates of his height ranged from 5'6 to 6'3, his hair was variously described as being brown, black, or red, and his weight was somewhere from 140 pounds to 230.
The information that the students retrieved from their short term memory, which had not yet been encoded as long term memory, was already profoundly unreliable.
I want to see the regression of LW karma (or the log of LW karma) on probability-Amanda-is-guilty!
You ought to leave Eliezer out of the equation, or assign him a Karma value equal to Yvain's, or else he'll dominate the regression.
Amazing job putting this together.
I had remembered LW as being a hive of Amanda fans, but either I’m succumbing to hindsight bias or people updated significantly after those articles.
My .13 probability in the first thread definitely went down following further discussion and, in particular, komponisto's second post. In the last year I've been comfortable using "She's definitely innocent" talking about the case with non-LWers.
I updated my 60% guilt for Knox/Sollecito almost immediately after reading the follow-up article. As I noted on that page, my 60% judgement was a clear case of anchoring. I started with the pro-guilt evidence and only managed an 80% guilt after reading only their evidence. That I was then only able to re-adjust down to 60% was absurd. Since I was (and still am) fairly weak as a rationalist, I probably should have withheld any kind of assessment until after I had read the pro-innocence website, rather than try to make an early assessment and update it w...
Strange thing about this is, if I've calculated it right, the average probability estimate of Guede's guilt is only ~87%. It seems to me that if this were your real probability estimate of his guilt, and you were on the jury at the guy's trial, you would be obligated to vote innocent. If you operate on the basis that a 13% chance of innocence is not a reasonable doubt, about thirteen out of every hundred people who go to jail will be innocent. That is (let me check) more than one in ten, which strikes me as rather a lot. I think my own estimate of Guede's guilt is above 99%, so I would vote guilty, but I'm surprised the average here is so low.
If you operate on the basis that a 13% chance of innocence is not a reasonable doubt, about thirteen out of every hundred people who go to jail will be innocent.
That's if everyone who went to jail had a 13% chance of innocence. Presumably much of the time it would be lower.
One person whose reflections would be particularly welcome, of course, is Rolf Nelson.
Our debate (which is currently stalled due to my fault -- it's my turn to reply) has dealt with one of the few important pieces of evidence to emerge after the original verdict: the incompatibility of the digestive evidence with the prosecution's hypothesized time of death. (This was covered during the original trial, but was never a focus of discussion among outside commentators until the folks at the JREF forum brought it to attention last year.)
Since I was randomly chosen to comment on this, I'll throw in my two cents. I haven't thought about too much and my first instinct was to trust whatever value judgements I had made at the time, which I thought were something like 5-5-95, but were actually 1-1-99. Since me-at-the-time was much more familiar than me-right-now, I'd still probably defer to his judgement; if anything, her exoneration and other evidence should move those numbers slightly closer to the extremes.
I assigned a relatively high probability of guilt for Amanda Knox because of a combination of ignorance and over-correction. I read material on the website of the innocent side first, and felt fairly convinced largely because of their assessment of the DNA evidence. The other website had a conflicting assessment of the DNA evidence, which I didn't know how to adjudicate without trying to learn more about how DNA evidence works. I didn't do this, and so remained uncertain but still thought that Knox was innocent with fairly high probability.
Additionally, I ...
My high estimate came from spending insufficient mental energy to come to a stable estimate as well as, as Eliezer said "if you assigned a probability higher than 15%, it meant you had a really major problem with crediting the opinions of other people and the authority of idiots."
So I'm a few months late to the game here, but I was one of the people selected to give my retrospective on this, so here goes:
My original estimates were .01, .01 and .99. I realize now that my calibration was off in the same way Komponisto has conceded his was: those numbers are way too strong to be rational estimates for something you read about on the internet for an hour or two. If I had it to do over again with benefit of hindsight, I'd probably say something more like .1 .1 .9.
The thing is though, in the time since then I've done quite a bit more re...
Hmm... I'm a bit confused about what was supposed to be predicted here. Were we supposed to predict whether Knox would be convicted, or predict whether Knox actually committed a murder? If I had been involved in the original conversations, I would have assigned a very low probability to Knox's actual guilt, but a higher probability to her being found guilty. One is a question that specifically pertains to Knox herself, and the other is a commentary on the state of the Italian justice system.
I can't remember exactly what I said in the last thread, but I think my opinion is basically the same now. I am reasonably confident that Knox and Sollecito were involved in the murder and very confident that Guede was.
I'm afraid this is a lesson for others to learn by observation and not one which you can learn yourself (without changing your mind). The reasoning goes along the lines:
Note that this is both an somewhat opposing but also complimentary lesson to the one Eliezer notes.
I vaguely recall that you got pretty annoyed at me a year or so ago when I pointed out a contradiction in your reasoning. I suspect that your anger at me over that incident is informing your commentary.
I've had no interaction with you on this site at all, but I have read your posts on the previous Amanda Knox threads, and while I believe I have a far greater aversion than wedrifid to making statements so likely to antagonize others, I have to say I find your judgment in this case in conjunction with your position as a lawyer downright frightening.
Judges are mostly selected from among lawyers...
At least in the United States, judges are mostly selected from among prosecutors. Defense attorneys, including public defenders, aren't very well represented on the bench. General judgment aside, this a serious systemic bias of the system.
Exactly what DNA evidence do you believe exonerates them?
The DNA evidence showing that Guede was the killer.
This is exactly the sort of thing that gets people exonerated under the Innocence Project. The only difference is, in most cases, the authorities don't usually go back to the scene desperately looking for new evidence to incriminate the original suspect.
(Er, not to sound vain, but my collaborator and I did a bit more than link to the report! ;-) )
Umm, does that mean yes or no?
For practical purposes, no.
Umm, does that mean yes or no?
It means that I do not strongly expect that there would be anything left on the scene which they would file as further evidence against Guede.
But when you "scour the scene," there's no way to narrow your search to DNA of a particular individual. Because the matching is determined back at the lab. Agree?
Yes and no. If they already have already established previously that person X has come into contact with objects X Y and Z, further testing of objects X Y and Z is likely to reveal further traces.
For the latter last two points, I do not have a strong expectation that further evidence against Guede would have been brought to court given that later retrieval would have been less reliable, and the case against him was already watertight, nor do I have a strong expectation that the defense would have claimed that Amanda Knox could not have been involved based on the weakness of the evidence of her involvement.
If we're going to go on though, let's review.
Meredith Kercher is killed. Knox and Sollecito are waiting outside when the police arrive to investigate the residence, and Edgarg...
I vaguely recall that you got pretty annoyed at me a year or so ago when I pointed out a contradiction in your reasoning.
I don't recall any conversations with you. (Mind you I expect I would have if I believed you then. Actually being wrong is embarrassing.)
No, from the premise "brazil84 is blatantly and obviously wrong despite paying attention to the topic" "don't do what brazil84 did" is a reasonably good inference to make. But as I noted you don't share that premise so naturally you should not be expected to believe it. This is why you were not the intended audience.
anonymous internet posters
Pseudonymous. There are many similarities, but having a long-standing name does have significant differences, even if the name isn't tied to one's "real-life" name.
So, to summarize why you didn't update:
Boy was Upton Sinclair ever right.
This comment is unrelated to the main article.
I take issue with your Bayes-prayer. I don't mind so much that it seems to be just a normal prayer with some replaced words, rather than being something good in its own right, though I think this would offend other LWers. However, it does violate one message on LW that I've found very important to internalize:
Never confess to me that you are just as flawed as I am unless you can tell me what you plan to do about it.
So, apparently, we're really really bad at bayescraft. What are we going to do about it?
I'd appreciate being removed as I don't feel like I spent much time reading or thinking before making my estimate and probably never should have commented. (Sorry for ruining your perfectly good and valid experiment...)
Edit: Actually, it doesn't look like I spent all that much less time on a log scale than other folks. It does seem like we didn't really have enough time to get a solid grip on things though.
Can you give a numerical probability? You claimed that your assessment had shifted somewhat in light of the information that Knox and Sollecito had been found not guilty on appeal.
I'm going to do a run down of my probability estimates for Knox, Sollecito, and Guede, given the evidence available to me excluding the argument I've already given for the unlikelihood of Knox and Sollecito's presence at the scene given the state of the forensic evidence. Could you explain what, if any of it, you disagree with, and how your own analysis would differ?
Prior probabilities. Common wisdom is that most people are killed by someone they know. I'm going to go with 80% likelihood of murder by an acquaintance, although I can't actually find a single figure all sources agree on. So we'll weight the prior probability of guilt such that people Kercher knows account for 80%, and everyone else accounts for 20%. Unfortunately, I can't find a good source estimating how many people most people know, and it depends how close a standard of acquaintance you're measuring. I'm going to give a ballpark estimate of about 1000, and thus apportion an average of .00125 prior likelihood of guilt between them.
For Knox, let's adjust up for closeness of acquaintance (we'll call it a factor of 10:1) for a likelihood of .0125. She had no prior arrest record; I'll adjust down by 70 percent to .00875, and she was a woman who was neither a close family member nor an intimate partner, so I'll adjust it down by another 90%, to .000875. I don't feel that the prosecution established any good reason to suspect a motive, but I'll be charitable to their case and not revise further down for that.
I'll call Sollecito's probability .0007, adjusting upwards for his increased likelihood as a male, but downwards based on his much lesser closeness to the victim; I have to take probability mass away from slight acquaintances if I'm adding it to close ones.
Guede was male, only slightly acquainted with Kercher, had been involved in breaking and entering prior to the case, and also did not have a clear motive. People with prior criminal records account for a majority of violent crimes despite being a minority of the population. The population without a criminal record is giving up a majority of their probability mass to the minority with, so this is a pretty significant boost. I don't know the proportion of Italy's population that has a record, but if we assume similar rates to the U.S. this will increase Guede's likelihood by about 11 times. So let's call his prior probability about .0077.
There's a considerable amount of fudge factor in these numbers, and you are of course free to take issue with them, but I think they give us a reasonable ballpark order to start with.
First, we'll deal with Guede. His handprint in blood places him conclusively at the scene of the crime (I doubt that anyone has ever left a handprint in blood at a scene at which they were not present) and he does not deny having been at the scene, so we'll call the odds of his presence infinitesimally below 1. He changed his story repeatedly over the course of his interrogation, beginning with "another anonymous black man did it while I was in the bathroom after having sex with her." The prior likelihood of this is very low, and Guede left DNA traces he was unlikely to have left if he was not involved in killing her. I'd adjust Guede up from the prior by an order of about 10^7, which would leave a likelihood of guilt that approximates to 1-(10^-5).
Let's move on to Amanda Knox. Giobbi first flagged her as a suspect based on the belief that her mannerisms at the crime scene were odd. I expect that a person is significantly more likely to behave oddly at a crime scene (relative to the norm for an uninvolved person) if they were involved than if they were uninvolved. I'd put an odds ratio of maybe 15 to 1 on it, except that Knox was already known to have odd mannerisms (such as doing yoga stretches in places considered inappropriate,) so I'd revise that down to maybe 5 to 1. So that gives us a probability of involvement for Knox of about .0044. All the evidence brought forward by the police after this point is filtered evidence, because they focused their investigations on Knox and Sollecito, and put more effort into finding reasons to suspect them than anyone else who might have been involved.
In the United States in 1976-1996, 16% of murders were committed by more than one person; since I don't have any better figures I'll assume similar rates for present day Italy. Ignoring the failure of forensics to turn up evidence of anyone else, I find the arguments posed by the prosecution for believing that the murder had multiple perpetrators extremely uncompelling. Arguments such as the number of stab wounds and the fact that Kercher had wounds on both sides of her neck are extremely weak evidence; particularly taken alongside claims such as one which they offered for the break-in being staged, that the pattern of the broken glass was inconsistent with a break from outside, which was tested and found to be false, these arguments sound to me like scrambling for anything that sounds like evidence. Ignoring the failure of anyone else to leave biological traces, I'd be willing to extend perhaps a 3 to 1 odds ratio on their evidence for multiple attackers, so together with the prior unlikelihood of a murder having multiple perpetrators, that brings Knox's likelihood of guilt to about .0022.
Knox and Sollecito failed to implicate Guede, and Guede failed to implicate Knox and Sollecito; for their mutual failure to implicate each other, I'd reduce Knox and Sollecito's likelihood about 20 times, bringing Knox to about .00011.
I do not find any evidence presented by the prosecution compelling with regards to establishing a motive or likelihood of presence at the crime for Knox or Sollecito, their case seems to me to be a collection of behaviors and actions attributed to Knox with the implication "this seems more unusual for a person who was not involved than a person who was." Since this evidence is filtered, I only find it reasonable to update on it to the extent that I believe the body of evidence is stronger than what I should expect for anyone with my current assignment of guilt subject to a similar police investigation. I think if you really looked and sincerely believed you had reason to find anything, you could find a substantial body of reasons to be suspicious of anyone. One of the most common ways that people err is in believing that a false proposition must have no valid evidence, and every argument for increasing your confidence in it must be refutable. I would have to assume that people I know are generally significantly weirder and more suspicious than most of the population to assign an odds ratio of even 10 to 1 on the prosecution's evidence, but my social circle probably is fairly odd, so I'll give them that, but I think I'm being generous.
All in all, that gives approximately .001 odds of guilt for Knox. Sollecito was tagged as a suspect mainly through association with her, so I would say that he is at most as likely as she is to have been involved.
Of course, there is plenty of uncertainty in these figures, and this does not portray the entirety of the relevant evidence even ignoring the improbability of other people involved in the murder not leaving biological traces (one particular piece of evidence I even left out deliberately, you can try and guess what,) but I think it reflects a reasonable probability estimate given that omission for an amount of evidence that a person can reasonably be expected to process without a substantial time commitment.
Can you give a numerical probability?
Perhaps about 30%.
Could you explain what, if any of it, you disagree with, and how your own analysis would differ?
I don't have the energy to do a point by point response, but I will point out what I think is the central flaw in your reasoning.
The flaw, it seems to me, is that you seem to dismiss the circumstantial evidence against Knox (and Sollecito) as "filtered evidence." There are two problems with this.
First, the evidence is not filtered in the sense that the only evidence we (or the jury) saw i...
Continuing my interest in tracking real-world predictions, I notice that the recent acquittal of Knox & Sollecito offers an interesting opportunity - specifically, many LessWrongers gave probabilities for guilt back in 2009 in komponisto’s 2 articles:
Both were interesting exercises, and it’s time to do a followup. Specifically, there are at least 3 new pieces of evidence to consider:
Point 2 particularly struck me (the press attributes much of the acquittal to the expert report, an acquittal I had not expected to succeed), but other people may find the other 2 points or unmentioned news more weighty.
2 Probabilities
I was curious how the consensus has changed, and so, in some spare time, I summoned all the Conscientiousness I could and compiled the following list of 54 entries based on those 2 articles’ comments (sometimes inferring specific probabilities and possibly missing probabilities given in hidden subthreads), where people listed probabilities for Knox’s guilt, Sollecito’s guilt, and Guede’s guilt:
It’s interesting how many people assign a high-probability to Knox being guilty; I had remembered LW as being a hive of Amanda fans, but either I’m succumbing to hindsight bias or people updated significantly after those articles. (For example, Eliezer says .15 is too high, but doesn’t seem otherwise especially convinced; and later one reads in Methods of Rationality that "[Hagrid] is the most blatantly innocent bystander to be convicted by the magical British legal system since Grindelwald's Confunding of Neville Chamberlain was pinned on Amanda Knox.")
EDIT: Jack graphed the probability against karma:
2.1 Outliers
If we look just at >41% (chosen to keep contacts manageable), we find 12 entries out of 54:
I have messaged each of them, asking them to comment here, describing if and how they have since updated, and any other thoughts they might have. (I have also messaged the first 12 commenters or so, chronologically, with <41% confidence in Knox’s guilt, with the same message.) The commenters:
AngryParsley / Cyan / Daniel_Burfoot / Eliezer_Yudkowsky / GreenRoot / John_Maxwell_IV / LauraABJ / Mario / Matt_Simpson / Morendil / Psychohistorian / Shalmanese / Threads / Unknowns / badger / bentarm / bgrah449 / bigjeff5 / brazil84 / dilaudid / jimmy / kodos96 / lordweiner27 / mattnewport / nerzhin / tut
I look forward to seeing their retrospectives, or indeed, anyone's retrospectives on the matter.