Over the summer, Eliezer suggested (approximately, I am repeating this from memory) the following method for making an important decision:
This was essentially the method I used in coming to my (probably slightly low) estimate of the probability that Knox and Sollecito were innocent. It just felt like they were innocent, and I saw essentially no reason to suspect they were guilty. I will note that the 'pro-guilt' site that komponisto linked to was just horribly devoid of anything that I might consider evidence (if anything, that site did more to convince me of Knox's innocence than the pro-innocence site), and I did spend probably about 10 minute trying to find some evidence that they had missed, but completely failed.
On a different not, as I said at the time, 0.95 and 0.05 were just proxies for "pretty damn sure" and "pretty damned unlikely" - I have very little idea what 5% probability fee...
This is a better summary of what I said than what I actually said, so I hereby declare your distorted version to be my true teaching.
Despite the fact that my opinion on the case has hardly changed at all, these posts -- and thinking about the case in general -- were a tremendous learning experience for me. Some of the lessons include:
Less Wrong is good at getting the right answer. Believe it or not, the strong survey consensus in favor of innocence -- prior to my second post -- came as something of a pleasant surprise to me. You don't find this in many other places, despite the fact that the case is a no-brainer. I had assumed there would be more wishy-washiness and probabilities close to 50% than there turned out to be. (There was some of this, but less than I expected.)
People in general are bad at getting the right answer. As shown by the original verdict, not to mention all the numerous pro-guilt commentators on internet forums and elsewhere. What's surprising about that? Not much, perhaps, but I would say that one thing that is important about it is that it shows that huge, glaring errors of judgement are not restricted to Far Mode. Even on a mundane question such as this, people are susceptible to strange cognitive biases that can severely distort their assessment of evidence.
Confidence should depend
See my added comment. I did not assign a probability of 15%. I said that if you assigned a probability higher than 15%, it meant you had a really major problem with crediting the opinions of other people and the authority of idiots. My probability that Knox and Sollecito were guilty was "that's privileging the hypothesis", i.e., "I see no real evidence in its favor so same as prior probability", i.e., "really damned' low".
That's a neat compact algorithm but this doesn't change the fact that it produces the wrong answer.
Again, 15% isn't the maximum of a range. It's a number that's not just "wrong" but "sufficiently wrong to imply you need to adjust your emotional makeup".
If you need a number for me, put in "<0.01". I wouldn't have bet $20,000 at 99-to-1 odds over it at the time of writing that first paragraph, but I'm not quite sure anymore that this really means my probability is >0.01, it's not like I'd have taken the bet the other way.
The Knox thread was one of the first steps in my getting interested in predictions in general. It was a slow process and is still ongoing, but it has had me spend time on various calibration exercises, on PredictionBook, on the Crowdcast instance dedicated to the Good Judgment project, on Inkling Markets because I saw a few arbitrage opportunities there that sounded like fun. I'm not as into predictions as gwern appears to be, but they're growing on me.
All that and I'm still not very sure what to think of the Knox case. Yes, if our predictions were being scored I'd be getting a non-trivial penalty from my 50% chance of her guilt - that is, if we take the outcome of the appeals process as an arbitration of the prediction, and judge, for the purposes of scoring, that she was "in fact" innocent. (I'm not saying I have much doubt now about her innocence: I'm saying that we won't ever know for sure, and part of the point of these prediction exercises is to allow us to better deal with that permanent uncertainty.)
On the other hand, some of the people listed above would be taking a much more serious hit. One thing I've learned from my various exercises is that you can't expect t...
Brain scans which can retrieve memories accurately?
I believe that the current understanding is that memory encoding, not just retrieval, is pretty unreliable, so even if you can read exactly what's in people's brains, it may not be much help.
My high school psychology teacher gave my class an interesting demonstration on the usefulness of eyewitnesses.
For one class, we found a notice on the door saying that the class had been relocated to another room. We went to that room, and shortly after our teacher arrived, followed by another teacher. He complained that she had not gotten proper clearance to move her class to that room, and he needed it for an exercise for his own class. They spent a couple minutes arguing, and harsh words were exchanged, after which he left the room.
Almost immediately after he left, our teacher asked us to create profiles of his physical description. Estimates of his height ranged from 5'6 to 6'3, his hair was variously described as being brown, black, or red, and his weight was somewhere from 140 pounds to 230.
The information that the students retrieved from their short term memory, which had not yet been encoded as long term memory, was already profoundly unreliable.
I want to see the regression of LW karma (or the log of LW karma) on probability-Amanda-is-guilty!
You ought to leave Eliezer out of the equation, or assign him a Karma value equal to Yvain's, or else he'll dominate the regression.
Amazing job putting this together.
I had remembered LW as being a hive of Amanda fans, but either I’m succumbing to hindsight bias or people updated significantly after those articles.
My .13 probability in the first thread definitely went down following further discussion and, in particular, komponisto's second post. In the last year I've been comfortable using "She's definitely innocent" talking about the case with non-LWers.
I updated my 60% guilt for Knox/Sollecito almost immediately after reading the follow-up article. As I noted on that page, my 60% judgement was a clear case of anchoring. I started with the pro-guilt evidence and only managed an 80% guilt after reading only their evidence. That I was then only able to re-adjust down to 60% was absurd. Since I was (and still am) fairly weak as a rationalist, I probably should have withheld any kind of assessment until after I had read the pro-innocence website, rather than try to make an early assessment and update it w...
Strange thing about this is, if I've calculated it right, the average probability estimate of Guede's guilt is only ~87%. It seems to me that if this were your real probability estimate of his guilt, and you were on the jury at the guy's trial, you would be obligated to vote innocent. If you operate on the basis that a 13% chance of innocence is not a reasonable doubt, about thirteen out of every hundred people who go to jail will be innocent. That is (let me check) more than one in ten, which strikes me as rather a lot. I think my own estimate of Guede's guilt is above 99%, so I would vote guilty, but I'm surprised the average here is so low.
If you operate on the basis that a 13% chance of innocence is not a reasonable doubt, about thirteen out of every hundred people who go to jail will be innocent.
That's if everyone who went to jail had a 13% chance of innocence. Presumably much of the time it would be lower.
One person whose reflections would be particularly welcome, of course, is Rolf Nelson.
Our debate (which is currently stalled due to my fault -- it's my turn to reply) has dealt with one of the few important pieces of evidence to emerge after the original verdict: the incompatibility of the digestive evidence with the prosecution's hypothesized time of death. (This was covered during the original trial, but was never a focus of discussion among outside commentators until the folks at the JREF forum brought it to attention last year.)
Since I was randomly chosen to comment on this, I'll throw in my two cents. I haven't thought about too much and my first instinct was to trust whatever value judgements I had made at the time, which I thought were something like 5-5-95, but were actually 1-1-99. Since me-at-the-time was much more familiar than me-right-now, I'd still probably defer to his judgement; if anything, her exoneration and other evidence should move those numbers slightly closer to the extremes.
I assigned a relatively high probability of guilt for Amanda Knox because of a combination of ignorance and over-correction. I read material on the website of the innocent side first, and felt fairly convinced largely because of their assessment of the DNA evidence. The other website had a conflicting assessment of the DNA evidence, which I didn't know how to adjudicate without trying to learn more about how DNA evidence works. I didn't do this, and so remained uncertain but still thought that Knox was innocent with fairly high probability.
Additionally, I ...
My high estimate came from spending insufficient mental energy to come to a stable estimate as well as, as Eliezer said "if you assigned a probability higher than 15%, it meant you had a really major problem with crediting the opinions of other people and the authority of idiots."
So I'm a few months late to the game here, but I was one of the people selected to give my retrospective on this, so here goes:
My original estimates were .01, .01 and .99. I realize now that my calibration was off in the same way Komponisto has conceded his was: those numbers are way too strong to be rational estimates for something you read about on the internet for an hour or two. If I had it to do over again with benefit of hindsight, I'd probably say something more like .1 .1 .9.
The thing is though, in the time since then I've done quite a bit more re...
Hmm... I'm a bit confused about what was supposed to be predicted here. Were we supposed to predict whether Knox would be convicted, or predict whether Knox actually committed a murder? If I had been involved in the original conversations, I would have assigned a very low probability to Knox's actual guilt, but a higher probability to her being found guilty. One is a question that specifically pertains to Knox herself, and the other is a commentary on the state of the Italian justice system.
I can't remember exactly what I said in the last thread, but I think my opinion is basically the same now. I am reasonably confident that Knox and Sollecito were involved in the murder and very confident that Guede was.
I'm afraid this is a lesson for others to learn by observation and not one which you can learn yourself (without changing your mind). The reasoning goes along the lines:
Note that this is both an somewhat opposing but also complimentary lesson to the one Eliezer notes.
I vaguely recall that you got pretty annoyed at me a year or so ago when I pointed out a contradiction in your reasoning. I suspect that your anger at me over that incident is informing your commentary.
I've had no interaction with you on this site at all, but I have read your posts on the previous Amanda Knox threads, and while I believe I have a far greater aversion than wedrifid to making statements so likely to antagonize others, I have to say I find your judgment in this case in conjunction with your position as a lawyer downright frightening.
Judges are mostly selected from among lawyers...
At least in the United States, judges are mostly selected from among prosecutors. Defense attorneys, including public defenders, aren't very well represented on the bench. General judgment aside, this a serious systemic bias of the system.
Exactly what DNA evidence do you believe exonerates them?
The DNA evidence showing that Guede was the killer.
This is exactly the sort of thing that gets people exonerated under the Innocence Project. The only difference is, in most cases, the authorities don't usually go back to the scene desperately looking for new evidence to incriminate the original suspect.
(Er, not to sound vain, but my collaborator and I did a bit more than link to the report! ;-) )
Umm, does that mean yes or no?
For practical purposes, no.
Umm, does that mean yes or no?
It means that I do not strongly expect that there would be anything left on the scene which they would file as further evidence against Guede.
But when you "scour the scene," there's no way to narrow your search to DNA of a particular individual. Because the matching is determined back at the lab. Agree?
Yes and no. If they already have already established previously that person X has come into contact with objects X Y and Z, further testing of objects X Y and Z is likely to reveal further traces.
For the latter last two points, I do not have a strong expectation that further evidence against Guede would have been brought to court given that later retrieval would have been less reliable, and the case against him was already watertight, nor do I have a strong expectation that the defense would have claimed that Amanda Knox could not have been involved based on the weakness of the evidence of her involvement.
If we're going to go on though, let's review.
Meredith Kercher is killed. Knox and Sollecito are waiting outside when the police arrive to investigate the residence, and Edgarg...
I vaguely recall that you got pretty annoyed at me a year or so ago when I pointed out a contradiction in your reasoning.
I don't recall any conversations with you. (Mind you I expect I would have if I believed you then. Actually being wrong is embarrassing.)
No, from the premise "brazil84 is blatantly and obviously wrong despite paying attention to the topic" "don't do what brazil84 did" is a reasonably good inference to make. But as I noted you don't share that premise so naturally you should not be expected to believe it. This is why you were not the intended audience.
anonymous internet posters
Pseudonymous. There are many similarities, but having a long-standing name does have significant differences, even if the name isn't tied to one's "real-life" name.
So, to summarize why you didn't update:
Boy was Upton Sinclair ever right.
This comment is unrelated to the main article.
I take issue with your Bayes-prayer. I don't mind so much that it seems to be just a normal prayer with some replaced words, rather than being something good in its own right, though I think this would offend other LWers. However, it does violate one message on LW that I've found very important to internalize:
Never confess to me that you are just as flawed as I am unless you can tell me what you plan to do about it.
So, apparently, we're really really bad at bayescraft. What are we going to do about it?
I'd appreciate being removed as I don't feel like I spent much time reading or thinking before making my estimate and probably never should have commented. (Sorry for ruining your perfectly good and valid experiment...)
Edit: Actually, it doesn't look like I spent all that much less time on a log scale than other folks. It does seem like we didn't really have enough time to get a solid grip on things though.
Continuing my interest in tracking real-world predictions, I notice that the recent acquittal of Knox & Sollecito offers an interesting opportunity - specifically, many LessWrongers gave probabilities for guilt back in 2009 in komponisto’s 2 articles:
Both were interesting exercises, and it’s time to do a followup. Specifically, there are at least 3 new pieces of evidence to consider:
Point 2 particularly struck me (the press attributes much of the acquittal to the expert report, an acquittal I had not expected to succeed), but other people may find the other 2 points or unmentioned news more weighty.
2 Probabilities
I was curious how the consensus has changed, and so, in some spare time, I summoned all the Conscientiousness I could and compiled the following list of 54 entries based on those 2 articles’ comments (sometimes inferring specific probabilities and possibly missing probabilities given in hidden subthreads), where people listed probabilities for Knox’s guilt, Sollecito’s guilt, and Guede’s guilt:
It’s interesting how many people assign a high-probability to Knox being guilty; I had remembered LW as being a hive of Amanda fans, but either I’m succumbing to hindsight bias or people updated significantly after those articles. (For example, Eliezer says .15 is too high, but doesn’t seem otherwise especially convinced; and later one reads in Methods of Rationality that "[Hagrid] is the most blatantly innocent bystander to be convicted by the magical British legal system since Grindelwald's Confunding of Neville Chamberlain was pinned on Amanda Knox.")
EDIT: Jack graphed the probability against karma:
2.1 Outliers
If we look just at >41% (chosen to keep contacts manageable), we find 12 entries out of 54:
I have messaged each of them, asking them to comment here, describing if and how they have since updated, and any other thoughts they might have. (I have also messaged the first 12 commenters or so, chronologically, with <41% confidence in Knox’s guilt, with the same message.) The commenters:
AngryParsley / Cyan / Daniel_Burfoot / Eliezer_Yudkowsky / GreenRoot / John_Maxwell_IV / LauraABJ / Mario / Matt_Simpson / Morendil / Psychohistorian / Shalmanese / Threads / Unknowns / badger / bentarm / bgrah449 / bigjeff5 / brazil84 / dilaudid / jimmy / kodos96 / lordweiner27 / mattnewport / nerzhin / tut
I look forward to seeing their retrospectives, or indeed, anyone's retrospectives on the matter.