Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

komponisto comments on The Amanda Knox Test: How an Hour on the Internet Beats a Year in the Courtroom - Less Wrong

42 Post author: komponisto 13 December 2009 04:16AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (632)

You are viewing a single comment's thread.

Comment author: komponisto 19 February 2010 02:17:02AM 5 points [-]

(Beginning thread for a debate with Rolf Nelson. Others also welcome to comment, of course.)

Okay, Rolf, so to get things started, I'd like to get your numbers out on the table. So, if you wouldn't mind, please tell me, first of all, your current posterior probability estimates for the guilt of:

  1. Amanda Knox
  2. Raffaele Sollecito
  3. Rudy Guede

(I expect we'll mainly focus on Knox and Sollecito, since that's obviously where our main disagreement is; I've included Guede for the sake of comparison.)

Next, I'd like to know your priors for Knox and Sollecito (and Guede as well, if you wish), by which I mean your estimate of the probability that each suspect would commit a crime of this sort, as of (let's say) a month before Kercher's death.

Then, if you could, please list, in descending order of evidentiary strength, the pieces of information most responsible for moving your estimates away from the above priors -- along with, if you wouldn't mind, a rough order-of-magnitude estimate of the strength of each piece of evidence, in terms of likelihood ratios, bits, or bels, whatever you prefer. (Again, I'm most interested in Knox and Sollecito; Guede is optional, at least for now.)

This should allow us to quickly pinpoint our disagreement(s).

Comment author: FAWS 22 February 2010 12:28:48AM *  2 points [-]

Oh, let me play!

(When you made your first post on this issue I found trying to look for unbiased information a terribly frustrating experience so I didn't look for more than 20 minutes, and haven't done any reading on it since, except for a cursory look the the wikipedia page just now. A list of all points that are agreed on by both sides (with sub-points arguing about the relevance of the point from both perspectives, perhaps) would have been very welcome)

Current posterior: not really sure, let's see what I end up with below, but as a starting point:

0.01<P(S) < P(K) <0.2 < 0.8< P (G) < 0.99

Priors for commiting a homicide in a specific month:

P(K)= 4.7 *10^-6 (US homicide rate, assuming being female and a young adult roughly cancel out)

P(S)= 4 *10^-6 (Italian homicide rate assuming young adult males are 4 times as likely to commit murder as average)

P(G)=1*10^-5 (had been implicated in a break-in)

An inhabitant of the top floor of that apartment being murdered in her room in this same specific month (R):

P(R|K)=0.1, P(R|~K)=4.5*10^-6

P(R|S)=0.04, P(R|~S)=4.95*10^-6

P(R|G)=0.0005, P(R|~G)=5*10^-6

Guede's DNA being found all over and inside the victim of a homicide in this same specific month (D_G):

P(DG|K)=1*10^-4, P(DG|~K)=6*10^-7

P(DG|S)=5*10^-5, P(DG|~S)=6*10^-7

P(DG|G)=0.6, P(DG|~G)=1*10^-7

I can't think of any other pieces of evidence that I can treat as effectively independent. Given R and treating the probability of any murders except R as effectively 0: Knox' DNA not being found on the victim (D_K):

P(DK|K)=0.5 P(DK|~K)=0.8

P(DK|S)==0.85 P(DK|~S)=0.81

P(DK|G)=0.82 P(DK|~G)=0.82

Sollecito's DNA being found on bra clasp of the the victim, but nowhere else (D_S):

P(DS|K)=0.0002 P(DS|~K)=0.00006

P(DS|S)==0.001 P(DS|~S)=0.00005

P(DS|G)=0.00007 P(DS|~G)=0.00007

Minimal trances of R's DNA found on the blade of one of the knifes in Sollecito's kitchen possibly matching one of three wounds, along with Knox' DNA on the handle (D_R):

P(DR|K)=5*10^-6 P(DR|~K)=1*10^-6

P(DR|S)=1*10^-5 P(DR|~S)=1*10^-6

P(DR|G)=1*10^-6 P(DR|~G)=1*10^-6

Trying to calculate the probabilities based on those estimates I find that I shouldn't have treated DG and R as independent either, I get a stupidly high result of 0.9999983 for P(G), and the indirect association with Guede shouldn't make the other two much more likely given that they are already more directly associated with the victim, if anything DG should make them less likely. The background probability for R should be higher because I forgot to properly account for the fact that there was more than 1 possible victim. Since Guede's DNA and being a room mate alone would already be enough to make Knox almost certainly guilty based on the numbers above and this makes no sense whatsoever I think we can safely say I thoroughly failed this test. I guess the lesson is that making up plausible numbers for various conditional probabilities well outside your intuitive range and then applying them in a basian update doesn't improve your calibration if you have no experience at it at all.

Comment author: komponisto 22 February 2010 02:15:28AM 3 points [-]

When you made your first post on this issue I found trying to look for unbiased information a terribly frustrating experience

One of the lessons of this exercise, that may be worth stating explicitly, is that there's no "outside referee" you can look to to make sure your beliefs are correct. In real life, you have to make judgments under uncertainty, using whatever evidence you have.

It's not as hard as you (and others) think. Yes, of course, the sources are "biased" in the sense that they have an incentive to mislead if they can get away with it. But what they say is not literally all the information you have. You also have background knowledge about how the world works. Priors matter. If A says X and B says ~X, and there's no a priori reason to trust one over the other, that doesn't mean you're stuck! It depends on how plausible X is in the first place.

I guess the lesson is that making up plausible numbers for various conditional probabilities well outside your intuitive range and then applying them in a basian update doesn't improve your calibration if you have no experience at it at all.

Here's the real lesson: Bayesian calculations are not some mysterious black-magic technique that you "apply" to a problem. They are supposed to represent the calculations your brain has already made. Probability theory is the mathematics of inference. If you have an opinion on this case, then, ipso facto, your brain has already performed a Bayesian update.

The mistake you made was not making up numbers; it was making up numbers that, as you point out in the end, didn't reflect your actual beliefs.

Comment author: FAWS 22 February 2010 03:11:44AM *  2 points [-]

One of the lessons of this exercise, that may be worth stating explicitly, is that there's no "outside referee" you can look to to make sure your beliefs are correct. In real life, you have to make judgments under uncertainty, using whatever evidence you have.

I meant that questions that should have an easily determinable answer, like "Did someone clean the blood outside her room up before the police was called" were unreasonably difficult to settle. Every site was mixing arguments and conclusions with facts. Sure, it's possible to find the answers if you look long enough, but it's much more work than it should be, and more work than I was willing to invest for a qestion that didn't interest me all that much in the first place.

Here's the real lesson: Bayesian calculations are not some mysterious black-magic technique that you "apply" to a problem. They are supposed to represent the calculations your brain has already made. Probability theory is the mathematics of inference. If you have an opinion on this case, then, ipso facto, your brain has already performed a Bayesian update.

The brain doesn't operate with very small or very big numbers, though. And I doubt it operates with conditional probabilities of the sort used above, as far as it operates Bayesian at all I would guess it's more similar to using venn diagrams.

The mistake you made was not making up numbers; it was making up numbers that, as you point out in the end, didn't reflect your actual beliefs.

The point is that I didn't spot that until after I did the calculation, and while I don't usually do much in the way of statistics I intuitively got the simple Bayesian problems like the usual example with false positives in a medical test right before hearing about Bayes theorem for the first time, so I don't think it's because I'm particularly bad at this. If you need to tweak afterwards anyway doing the Bayesian update explicitly isn't very useful as self-control.

Comment author: AnnaGilmour 21 February 2010 08:12:26PM *  0 points [-]

This should allow us to quickly pinpoint our disagreement(s).

The disagreement most likely stems from the reliability of the Micheli Report for accuracy and comprehensiveness.

Comment author: AnnaGilmour 21 February 2010 08:08:33PM 0 points [-]

Posterior probability estimates:

  1. 0
  2. 0
  3. .9

Priors:

  1. .01
  2. .01
  3. .5

Is that the sort of thing you are asking? I don't know if I attributed correctly.

Comment author: komponisto 21 February 2010 08:22:31PM *  0 points [-]

Anna: for the context of this, see here.

You may want to remember that 0 and 1 are not probabilities. Also, I must say I don't understand your extremely high prior of 0.5 for Guede. (The evidence against him is such that the prior could be much, much lower and he would still have a very high probability of guilt.)

Comment author: AnnaGilmour 23 February 2010 01:52:42AM 2 points [-]

The thing that I am trying to point out is that I believe Amanda and Raffaele were wrongly included in the class called "suspects".

Comment author: komponisto 23 February 2010 02:11:41AM 1 point [-]

Yes indeed -- our term for that here is privileging the hypothesis.

Comment author: wedrifid 23 February 2010 02:13:39AM 0 points [-]

(Although I do find the point more salient when it is described explicitly rather than by reference to jargon. )

Comment author: komponisto 23 February 2010 02:34:20AM 1 point [-]

Wasn't trying to enforce the use of jargon so much as classify the fallacy.

After all, the point is even more salient when you can relate it to a whole category of error found in many other contexts.

Comment author: AnnaGilmour 23 February 2010 04:44:10AM *  1 point [-]

I find the term useful. I think it is what a lot of the media has done. Since Amanda and Raffaele are in discussion and named in the theory, there must be something to it and they have equal weights of measure for concern as the third suspect, Rudy. When in fact, they are very lightweight and the (heavy) weight should be attributed to the method by which they became suspects. The term helps me to say "Oh that's what is going on." Like komponisto said, a whole category of error. (Not to mention all the contexts apart from this specific case, the topic at hand, indeed.)

Comment author: komponisto 23 February 2010 05:01:13AM 3 points [-]

Quite right. It's actually amazing how little attention was paid to Rudy Guede in the media coverage of this case, particularly in the U.S. and U.K. media. (Numerous stories either omitted all mention of him altogether or else referred briefly and vaguely to "a third suspect" -- without any hint about the disparity in evidence.)

Comment author: AnnaGilmour 23 February 2010 01:58:36AM 0 points [-]

I am also pointing out that is a question pertaining to applied situation with a limited scope - the decision to convict or exonerate. For all intents and purposes, relative "knowing" is permissible in a legal case, since we are dealing with human events and activities of a finite nature - a court decision is a discrete (not continuous) matter. After a certain point, probabilties have to turn into decisions.

Comment author: AnnaGilmour 23 February 2010 02:02:12AM -2 points [-]

Therefore, I offered 0 in the spirit of Goedel's completeness theorem, yes, at the expense of consistency. Consistency will yield a perpetual motion situation. Completeness is required and can be appropriately reached through reasoning, logic, objectivity, etc. Something can only consistent OR complete. Not both.

Comment author: AnnaGilmour 21 February 2010 08:45:07PM *  0 points [-]

In other words, his psychological profile and actions leading up to the murder do not indicate that he was above board and immune from a violent attack, especially an attack with a knife. He was also known around town to go too far in the direction toward harassment of females around town at the clubs and so forth. He was also known to do various drugs including aggression-increasing drugs such as cocaine. He was known to break and enter and steal, and that he carried a ten inch knife "for protection" (his words). It could be argued then that it was a matter of brief time for him to break and enter, steal, and encounter someone indoors in the process as was arguably such in the situation with Meredith, and "defend" himself when caught or interrupted. This is the case that I would start to make as for a high prior.

Comment author: AnnaGilmour 21 February 2010 08:40:25PM 0 points [-]

It is true that a high probability of a prior is not necessary for probability of guilt.

It is also true, however, that it doesn't mean that he didn't have a high prior. I could drop it to .3 though. With the actions in the previous weeks, a case could be made that he was in an escalating pattern of behavior, which is why I gave him a .5 prior.

Comment author: AnnaGilmour 21 February 2010 08:36:49PM 0 points [-]

Thank you. Yes I've seen the post by Rolf Nelson.

I don't understand (though I admit for expediency's sake did not fully read the 0/1 link which I should do if I post here) how there cannot be an absolute for innocence. I didn't assign 1 to Rudy Guede for the reason you mention. But in terms of innocence, we know for example that Princess Diana didn't kill Meredith and that the mayor of Seattle at the time did not kill Meredith, so how can it not be zero? I wrote zero for a specific reason. I wanted it to indicate that gap between reasonability of arrest and no reasonability of arrest. To assign even a small possibility at this point seems inaccurate to me. Although, you make a good point, in actuality, so I would amend them to .001. Is that a proper probability quotient in terms of the question?

Comment author: komponisto 21 February 2010 08:42:53PM 0 points [-]

Although, you make a good point, in actuality, so I would amend them to .001. Is that a proper probability quotient in terms of the question?

Yes, that would be more reasonable (indeed, it's about where my own estimate is).

Comment author: AnnaGilmour 21 February 2010 11:24:02PM 0 points [-]

Is it possible to show that it would be impossible for them to have been participants making it 0? Is there anyone in the world in that class - of 0? Trying to understand the parameters of "probability".

Comment author: RobinZ 21 February 2010 11:55:35PM 3 points [-]

I agree with Alicorn - a probability of 0 or 1 can only be legitimately used as hyperbole.

(There's a technical explanation for why to exclude probabilities of 0 and 1, but it assumes you have studied and understood Bayes theorem and know how to think about probabilities in real life in Bayes terms.)

Comment author: Alicorn 21 February 2010 11:27:37PM *  3 points [-]

Exactly-0 isn't on the table at all. Close-enough-to-0-that-you-can-represent-it-that-way-without-too-much-disclaiming is reserved for propositions like "a square circle and Batman teamed up to, not kill, but kidnap and replace with a convincing inert android, Meredith". Princess Diana's odds of having killed Meredith are miniscule, but not zero or even compellingly zerolike, compared to those.

Comment author: Jack 22 February 2010 12:23:36AM 1 point [-]

"a square circle".

I don't know if this means I disagree with Eliezer but I'm pretty sure the probability of a contradiction has to be 0 and the probability of a tautology has to be 1. Else really weird things start happening and you can't do deduction. Like, what is the probability of A given A ^ B?

Comment author: RobinZ 22 February 2010 12:32:51AM *  11 points [-]

*cackles evilly and cracks metaphorical knuckles*

The circle is defined as the locus of points an equal distance from a center on a plane. A square is defined as a regular quadrilateral - i.e. a shape with four sides of equal length separated by four angles of equal magnitude. If you allow that "distance" may be generalized to be applicable to other geometries than Euclidean...

...what is the shape of a circle on a chessboard, where "distance" is measured by the number of king-moves?

I believe this is a useful object lesson in the difficulty of constructing properly impossible propositions.

Edited to make the square have four sides, not three. What was I thinking...?

Comment author: Jack 22 February 2010 12:42:34AM *  3 points [-]

And when you superimpose a middle finger onto Reimannian space...

Edit: But upvoted because it is always good to get this reminder.

Comment author: Alicorn 22 February 2010 12:39:55AM 4 points [-]

Something can be metaphysically/logically impossible without it being okay to assign exactly-0 to it. Epistemic probability is what we're really representing here - I mean, even something as uncertain-to-me as the current weather conditions in the red spot on Jupiter is exactly one way. But it's not useful to represent that single-ness of weather conditions because I can't access them. I similarly can't usefully access absolute epistemic certainty about even simple math and logic. I'm a broken machine; I cannot handle perfect surety.

Comment author: Jack 22 February 2010 03:08:53AM 1 point [-]

It isn't that simple. Most of the results we get from Bayes theorem we get by deduction. For example, the Dutch book argument, the most common justification given for Bayesian epistemology in the first place, relies on deduction. So does nearly every other important result we get from Bayes theorem. So when you say to someone: take this evidence and act rationally that may imply that that person not get her deductions wrong. This is why, afaict most Bayesians assume logical omniscience. See here. Apparently there have been attempts to weaken logical omniscience and maybe someone here has one in mind... but I haven't heard it. Obviously it is that case the humans, as a matter of psychological fact can screw up deduction. But this is basically like saying that as a matter of psychological fact humans aren't perfect Bayesian rationalists. The whole theory isn't supposed to be descriptive, it is an ideal to strive toward.

Comment author: wedrifid 22 February 2010 01:17:51AM 0 points [-]

I have also seen Eliezer tempted to consider a '0' probability in response to a 'divide by infinity' situation. (I think there is a 'mathsy' way to represent that kind of '0'.)

Comment author: kpreid 22 February 2010 01:53:09AM 0 points [-]

That's called a limit. What's special is not the “zero” but the “infinity”: you don't talk about a value “infinity” (attempting to have one causes you to lose various other useful properties), but rather that as some input increases without bound, the output approaches zero.

“The limit of 1/x as x approaches infinity is zero."

Comment author: AnnaGilmour 21 February 2010 11:38:00PM 0 points [-]

Well, it is quite fascinating that no one gets a 0 probability. Just to ask, does Meredith get a 0 probability? I will move past understanding the exclusion of 0. I just want to make sure I understand. Anyway, when I say 0, I understand it to mean functionally 0, which is the same as .0000000001, which is also functionally 0, correct? Thank you for you patience.

Comment author: Alicorn 22 February 2010 12:34:53AM 5 points [-]

Meredith could have committed suicide. She's probably more likely to be responsible for the death than Princess Diana. And she's much more likely than the team of Batman-and-square-circle.

Comment author: AnnaGilmour 22 February 2010 01:02:56AM 0 points [-]

Were there any fatal wounds that she could not have inflicted?