Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Amanda Knox: post mortem

22 Post author: gwern 20 October 2011 04:10PM

Continuing my interest in tracking real-world predictions, I notice that the recent acquittal of Knox & Sollecito offers an interesting opportunity - specifically, many LessWrongers gave probabilities for guilt back in 2009 in komponisto’s 2 articles:

Both were interesting exercises, and it’s time to do a followup. Specifically, there are at least 3 new pieces of evidence to consider:

  1. the failure of any damning or especially relevant evidence to surface in the ~2 years since (see also: the hope function)
  2. the independent experts’ report on the DNA evidence
  3. the freeing of Knox & Sollecito, and continued imprisonment of Rudy Guede (with reduced sentence)

Point 2 particularly struck me (the press attributes much of the acquittal to the expert report, an acquittal I had not expected to succeed), but other people may find the other 2 points or unmentioned news more weighty.

2 Probabilities

I was curious how the consensus has changed, and so, in some spare time, I summoned all the Conscientiousness I could and compiled the following list of 54 entries based on those 2 articles’ comments (sometimes inferring specific probabilities and possibly missing probabilities given in hidden subthreads), where people listed probabilities for Knox’s guilt, Sollecito’s guilt, and Guede’s guilt:

Knox Sollecito Guede LWer
.20 .20 .70 badger
.05 .10 .90 mattnewport
.20 .25 .90 AngryParsley
.05 .05 .95 tut
.05 .05 .95 bentarm
.85 .60 .20 bgrah449
.01 .01 .99 kodos96
.01 .01 .99 Daniel_Burfoot
.40 .40 .90 nerzhin
.45 .45 .60 Matt_Simpson
.33 .33 .90 Cyan
.50 .50 .95 jimmy
.05 .05 .99 Psychohistorian
.40 .40 .90 Threads
.50 .50 .80 Morendil
.15 Eliezer_Yudkowsky
.20 .35 .98 LauraABJ
.10 .10 .90 curious
.20 .20 .96 jpet
.06 .06 .70 saliency
.80 .60 .95 Mario
.20 .20 .95 Yvain
.70 Shalmanese
.05 .05 .95 gelisam
.05 .05 .90 Mononofu
.90 .90 .90 lordweiner27 (changed mind)
.50 .50 .99 GreenRoot
.99 .99 .99 dilaudid
.13 .15 .97 Jack
.05 .05 .90 wedrifid
.01 .01 .90 Nanani
.35 .35 .95 imaxwell
.01 .01 .99 jenmarie
.25 .25 .75 Jawaka
.41 .38 .99 magfrump
.40 .20 .60 gwern
.08 .10 .95 loqi
.25 .25 .50 JamesAndrix
.90 .85 .99 Unknowns
.35 .35 .90 Sebastian_Hagen
.90 .90 .99 brazil84
.30 .30 .40 ChrisHibbert
.02 .02 .98 wnoise
.50 .40 .90 John_Maxwell_IV
.10 .10 k3nt
.01 .01 .99 Sinai
.00 .00 1.0 KayPea
.00 .00 .60 MerleRideout
.15 .10 .80 TheRev
.01 .01 .99 komponisto
.30 pete22
.01 SforSingularity
.00 .00 .90 AnnaGilmour
.05 .05 .95 Seth_Goldin
.60 .60 .95 bigjeff5

It’s interesting how many people assign a high-probability to Knox being guilty; I had remembered LW as being a hive of Amanda fans, but either I’m succumbing to hindsight bias or people updated significantly after those articles. (For example, Eliezer says .15 is too high, but doesn’t seem otherwise especially convinced; and later one reads in Methods of Rationality that "[Hagrid] is the most blatantly innocent bystander to be convicted by the magical British legal system since Grindelwald's Confunding of Neville Chamberlain was pinned on Amanda Knox.")

EDIT: Jack graphed the probability against karma:

2.1 Outliers

If we look just at >41% (chosen to keep contacts manageable), we find 12 entries out of 54:

Knox Sollecito Guede LWer
.45 .45 .60 Matt_Simpson
.50 .40 .90 John_Maxwell_IV
.50 .50 .80 Morendil
.50 .50 .95 jimmy
.50 .50 .99 GreenRoot
.60 .60 .95 bigjeff5
.70 Shalmanese
.80 .60 .95 Mario
.85 .60 .20 bgrah449
.90 .85 .99 Unknowns
.90 .90 .90 lordweiner27
.90 .90 .99 brazil84
.99 .99 .99 dilaudid

I have messaged each of them, asking them to comment here, describing if and how they have since updated, and any other thoughts they might have. (I have also messaged the first 12 commenters or so, chronologically, with <41% confidence in Knox’s guilt, with the same message.) The commenters:

AngryParsley / Cyan / Daniel_Burfoot / Eliezer_Yudkowsky / GreenRoot / John_Maxwell_IV / LauraABJ / Mario / Matt_Simpson / Morendil / Psychohistorian / Shalmanese / Threads / Unknowns / badger / bentarm / bgrah449 / bigjeff5 / brazil84 / dilaudid / jimmy / kodos96 / lordweiner27 / mattnewport / nerzhin / tut

I look forward to seeing their retrospectives, or indeed, anyone's retrospectives on the matter.

Allknowing and most merciful Bayes;
We have erred, and strayed from thy ways like biased sheep.
We have followed too much the devices and desires of our own hearts.
We have offended against thy axiomatic laws.
We have left undone those updates which we ought to have done;
And we have done those updates which we ought not to have done;
And there is no calibration in us.
But thou, O Bayes, have mercy upon us, miserable wannabes.
Spare thou them, O Bayes, who confess their faults.

Comments (483)

Comment author: bentarm 21 October 2011 11:24:16AM 26 points [-]

Over the summer, Eliezer suggested (approximately, I am repeating this from memory) the following method for making an important decision:

  1. write down a list of all of the relevant facts on either side of the argument.
  2. assign numerical weights to each of the facts, according to how much they point you in one direction or another.
  3. burn the piece of paper on which you wrote down the facts, and go with your gut.

This was essentially the method I used in coming to my (probably slightly low) estimate of the probability that Knox and Sollecito were innocent. It just felt like they were innocent, and I saw essentially no reason to suspect they were guilty. I will note that the 'pro-guilt' site that komponisto linked to was just horribly devoid of anything that I might consider evidence (if anything, that site did more to convince me of Knox's innocence than the pro-innocence site), and I did spend probably about 10 minute trying to find some evidence that they had missed, but completely failed.

On a different not, as I said at the time, 0.95 and 0.05 were just proxies for "pretty damn sure" and "pretty damned unlikely" - I have very little idea what 5% probability feels like, and I'm sure that if arbitrary scientific convention had settled on some different number for significance, I'd have picked that one instead. I have made some progress since a year ago on calibrating my estimates of small probabilities, but I absolutely do not think that I would be wrong approximately 1 time in 20 when making predictions to which I assign a probability of 0.95.

Comment author: Eliezer_Yudkowsky 21 October 2011 03:35:05PM 30 points [-]

This is a better summary of what I said than what I actually said, so I hereby declare your distorted version to be my true teaching.

Comment author: lessdazed 21 October 2011 05:39:53PM 11 points [-]

I have very little idea what 5% probability feels like

???

1d20!!!

Comment author: FiftyTwo 24 October 2011 10:13:21PM 2 points [-]

I don't have an intuitive feeling for d20s, but it occurs to me that a useful resource might be a list of day to day events of certain probabilities so we can calibrate our intuitions to them.

Googling hasn't found me anything useful, could anyone give an example of an normal event that has a 5% chance of occuring?

Comment author: saturn 24 October 2011 11:46:44PM 6 points [-]

You look at a clock and the seconds are :00, :01, or :02.

Comment author: JoshuaZ 25 October 2011 12:43:39AM *  1 point [-]

It is a little over the chance that if you are dealt two cards from a standard deck of cards that one of them will be the ace of spaces. It is a little under the chance that if you are dealt three cards from a standard deck that one them will be the ace of spades.

It is roughly the chance that if you pick three random members of the US House of Representatives that at least two of the three will not be reelected.

If It is about half as likely as the chance that a given US soldier in Iraq over the last decade will have been killed or too badly injured to return to duty (generally estimated to be around 9%). ETA: This number is wildly off. Disregard.

It is slightly less likely than your expectation for Schrodinger's cat to be alive if you run the experiment 5 times.

It is a bit under the chance that if you put your money on two numbers on a roulette wheel that one of them will turn up.

It is slightly over the chance that if you meet two random South Koreans that their last names will both be "Kim".

ETA: Here's a depressing one: It is around the chance that if you pick two children with childhood leukemia that they will both survive five years.

Comment author: lessdazed 25 October 2011 12:59:46AM 1 point [-]

It is about half as likely as a US soldier in Iraq over the last decade will be killed or too badly injured to return to duty (generally estimated to be around 9%).

Who exactly?

"One is always," said Bertrand, "a shirker to some one else."

"That's true; no matter what you call yourself, you'll always---always--find worse blackguards and better blackguards than yourself."

"All those that never go up to the trenches, or those who never go into the first line, and even those who only go there now and then, they're shirkers, if you like to call 'em so, and you'd see how many there are if they only gave stripes to the real fighters."

"There are two hundred and fifty to each regiment of two battalions," said Cocon.

"There are the orderlies, and a bit since there were even the servants of the adjutants."--"The cooks and the under-cooks."--"The sergeant-majors, and the quartermaster-sergeants, as often as not."--"The mess corporals and the mess fatigues."--"Some office-props and the guard of the colors."--"The baggage-masters." "The drivers, the laborers, and all the section, with all its non-coms., and even the sappers."--"The cyclists." "Not all of them."--"Nearly all the Red Cross service."--"Not the stretcher-bearers, of course; for they've not only got a devilish rotten job, but they live with the companies, and when attacks are on they charge with their stretchers; but the hospital attendants."

"Nearly all parsons, especially at the rear. For, you know, parsons with knapsacks on, I haven't seen a devil of a lot of 'em, have you?"

"Nor me either. In the papers, but not here."

"There are some, it seems."--"Ah!"

"Anyway, the common soldier's taken something on in this war."

"There are others that are in the open. We're not the only ones."

"We are!" said Tulacque, sharply; "we're almost the only ones!"

He added, "You may say--I know well enough what you'll tell me--that it was the motor lorries and the heavy artillery that brought it off at Verdun. It's true, but they've got a soft job all the same by the side of us. We're always in danger, against their once, and we've got the bullets and the bombs, too, that they haven't. The heavy artillery reared rabbits near their dug-outs, and they've been making themselves omelettes for eighteen months. We are really in danger. Those that only get a bit of it, or only once, aren't in it at all. Otherwise, everybody would be. The nursemaid strolling the streets of Paris would be, too, since there are the Taubes and the Zeppelins, as that pudding-head said that the pal was talking about just now."

"In the first expedition to the Dardanelles, there was actually a chemist wounded by a shell. You don't believe me, but it's true all the same--an officer with green facings, wounded!"

"That's chance, as I wrote to Mangouste, driver of a remount horse for the section, that got wounded--but it was done by a motor lorry."

"That's it, it's like that. After all, a bomb can tumble down on a pavement, in Paris or in Bordeaux."

"Oui, oui; so it's too easy to say, 'Don't let's make distinctions in danger!' Wait a bit. Since the beginning, there are some of those others who've got killed by an unlucky chance; among us there are some that are still alive by a lucky chance. It isn't the same thing, that, seeing that when you're dead, it's for a long time."

--Le Feu (Under Fire), translation.

Comment author: JoshuaZ 25 October 2011 01:12:15AM 2 points [-]

Who exactly?

Oh. Hmm. I don't remember where I saw this but that number is my background fact set. But when I look at the actual numbers this is clearly false. There have been around 30,000 people wounded or killed. (Source) and around a million who have served. That means that the probability of being wounded or killed at all is around .03, which is much smaller, and that's even before the fact that I said wounded severely enough that one can't keep fighting. Also in retrospect my number was obviously too high. Severe failure of rationality on my part. Ugh.

Comment author: komponisto 25 October 2011 12:44:45PM 0 points [-]

I thought that number was highly suspicious, but I attributed it to the combined category (killed or too injured to return -- which of course are very different things from the perspective of the individual concerned!).

Comment author: dlthomas 25 October 2011 12:09:15AM 1 point [-]

It's somewhere between the chance of flipping 4 successive heads and the chance of flipping 5 successive heads with a fair coin.

Comment author: thomblake 24 October 2011 11:50:23PM 1 point [-]

I was going to respond for I thought I knew many such things, but the few that did not involve rolling d20s involved rolling d%.

Comment author: wedrifid 21 October 2011 05:32:35PM 0 points [-]

I have made some progress since a year ago on calibrating my estimates of small probabilities, but I absolutely do not think that I would be wrong approximately 1 time in 20 when making predictions to which I assign a probability of 0.95.

My guess would be more that 1 in 20 wrong for a 95% confidence.

Comment author: komponisto 21 October 2011 02:16:15PM 15 points [-]

Despite the fact that my opinion on the case has hardly changed at all, these posts -- and thinking about the case in general -- were a tremendous learning experience for me. Some of the lessons include:

  • Less Wrong is good at getting the right answer. Believe it or not, the strong survey consensus in favor of innocence -- prior to my second post -- came as something of a pleasant surprise to me. You don't find this in many other places, despite the fact that the case is a no-brainer. I had assumed there would be more wishy-washiness and probabilities close to 50% than there turned out to be. (There was some of this, but less than I expected.)

  • People in general are bad at getting the right answer. As shown by the original verdict, not to mention all the numerous pro-guilt commentators on internet forums and elsewhere. What's surprising about that? Not much, perhaps, but I would say that one thing that is important about it is that it shows that huge, glaring errors of judgement are not restricted to Far Mode. Even on a mundane question such as this, people are susceptible to strange cognitive biases that can severely distort their assessment of evidence.

  • Confidence should depend on how much you know. I erred twice during the original sequence of posts on the case: once when I implied that participants in my survey ought to have come up with numbers like mine (0.001), and then again when I reacted to the indignation that followed by downgrading my own confidence. In truth, because (even before the verdict) I had followed the case more than most LW readers, my level of confidence should have been higher than most readers'. For someone who spends ten minutes reading the two sites I linked to, 0.9 probability of innocence (as opposed to 0.99) is perfectly reasonable; whereas someone like me should expect to be more confident, even if only on account of the meta-information that any additional information not on those sites isn't particularly relevant.

(More comments possibly to be added later...)

Comment author: wedrifid 21 October 2011 02:49:13PM 2 points [-]

What's surprising about that? Not much, perhaps, but I would say that one thing that is important about it is that it shows that huge, glaring errors of judgement are not restricted to Far Mode.

To the extent that I use that nomenclature I would have called this judgement to be a far mode one. It is the throwing about of far mode political abstractions to achieve perceived near mode goals. Those near mode goals have very little to do with the guilt or innocence of the victim (Amanda) and a lot to do with how your political utterance ("She's a witch! Burn her!") will be perceived by your peers.

Comment author: komponisto 21 October 2011 03:02:20PM 3 points [-]

Yes, we seem to have quite different understandings of what these terms ("near", "far", "political") mean.

An example of (erroneous) "near" reasoning in my usage would be: "Amanda is guilty because there had to be multiple attackers because there were so many wounds on the victim".

Whereas an example of (erroneous) "far" reasoning would be: "Amanda is guilty because f**ck those arrogant imperialist Americans trying to tell us how to run our country".

Comment author: Jack 21 October 2011 04:17:38PM 0 points [-]

In truth, because (even before the verdict) I had followed the case more than most LW readers, my level of confidence should have been higher than most readers'.

True while we're uncertain of your rationality. But at this point I find you reliable enough to think that your confidence is what mine would be if I followed the case as closely as you. And that means I'm just going to adopt your probability estimate.

Comment author: komponisto 21 October 2011 04:38:13PM *  1 point [-]

Well, of course, at the time in question no one knew what my probability estimate was; I merely meant that they need not have reached 0.999 confidence from a few minutes of browsing.

(Thanks for the compliment, in any case!)

Comment author: Morendil 20 October 2011 05:00:44PM 8 points [-]

The Knox thread was one of the first steps in my getting interested in predictions in general. It was a slow process and is still ongoing, but it has had me spend time on various calibration exercises, on PredictionBook, on the Crowdcast instance dedicated to the Good Judgment project, on Inkling Markets because I saw a few arbitrage opportunities there that sounded like fun. I'm not as into predictions as gwern appears to be, but they're growing on me.

All that and I'm still not very sure what to think of the Knox case. Yes, if our predictions were being scored I'd be getting a non-trivial penalty from my 50% chance of her guilt - that is, if we take the outcome of the appeals process as an arbitration of the prediction, and judge, for the purposes of scoring, that she was "in fact" innocent. (I'm not saying I have much doubt now about her innocence: I'm saying that we won't ever know for sure, and part of the point of these prediction exercises is to allow us to better deal with that permanent uncertainty.)

On the other hand, some of the people listed above would be taking a much more serious hit. One thing I've learned from my various exercises is that you can't expect to be right all the time - sometimes, with minimal knowledge of the relevant facts, a 50% prediction is in fact not so bad.

Then again, some of us were also apparently very confident in the answer that "in fact" turned out to be the correct one. Then again, we all get lucky from time to time - that too is the nature of the beast...

My intention is to continue to learn, to continue to get better at predicting, to become better calibrated and more discriminating over time.

Comment author: gwern 20 October 2011 05:12:02PM 5 points [-]

(I'm not saying I have much doubt now about her innocence: I'm saying that we won't ever know for sure, and part of the point of these prediction exercises is to allow us to better deal with that permanent uncertainty.)

One of the unfortunate things about living when we do is that it seems unlikely there will be any future oracles developed which reveal definitive answers to ancient crimes.

I refer, of course, to DNA evidence, which gave us an astonishing oracle to ask questions about old crimes, revealing a shockingly high lower bound on the justice system's error rates. If we had been around and recorded predictions about various death row inmates, then the Innocence Project's &etc. results would've been an assessment of our calibration worth writing home about!

Comment author: NancyLebovitz 20 October 2011 06:58:59PM 3 points [-]

One of the unfortunate things about living when we do is that it seems unlikely there will be any future oracles developed which reveal definitive answers to ancient crimes.

I wouldn't give that an extremely high probability. One of the trends in science is getting surprising amounts of information from tiny amounts of input-- DNA is one example, and finding extrasolar planets is another. I don't have specifics in mind, but I wouldn't be surprised if another method or two which are at least as powerful are developed.

Comment author: gwern 20 October 2011 07:13:53PM 3 points [-]

My reasoning goes along the lines of fingerprints and then DNA are highly precise near unique identifiers, which are also pretty sturdy and accurate. I don't know of any more biological traces which could significantly improve, much less be orders of magnitude superior to what went before like fingerprints & then DNA were.

There are plenty of future improvements in crime-solving, sure - lifelogging and pervasive surveillance comes to mind as the most obvious improvement. But none of the ones I can think of will be oracles in the sense I mean here of giving us the correct answers for cases we already 'solved', none of them will be retrospective. (Lifelogging will be employed as soon as available, witness the Canadian thing with stitching together hundreds of photos online to identify & arrest scores of rioters; I would be surprised if huge archives of recording built up and then only decades later are suddenly made public and cold case units began cracking cases with them, for example.)

Comment author: NancyLebovitz 20 October 2011 08:16:08PM 1 point [-]

Brain scans which can retrieve memories accurately? Admittedly, this would be limited to crimes with living witnesses.

The thing is, I think science leads to weird, surprising discoveries. I'm not going to predict what's impossible a century from now.

Comment author: Desrtopa 20 October 2011 10:37:39PM 9 points [-]

Brain scans which can retrieve memories accurately?

I believe that the current understanding is that memory encoding, not just retrieval, is pretty unreliable, so even if you can read exactly what's in people's brains, it may not be much help.

Comment author: gwern 20 October 2011 10:46:33PM 1 point [-]

And then there's the problem of locating the right person to interrogate - a task equivalent to and as hard as locating the hypothesis. :) But I probably shouldn't be too dismissive: brain-scanning is one of the better contenders for the next oracle.

Comment author: Desrtopa 20 October 2011 10:54:46PM *  4 points [-]

Well, having the right technologies can certainly make locating the hypothesis a lot easier; think how much harder it would have been to locate Guede as a suspect without DNA testing.

If we had a reliable way of determining self perceived truth value, nearly all interrogation could be narrowed down to "Do you know who did it?" "Who did it?" and "Did you do it?"

On second thought, given a device that were capable of doing that with a negligible failure rate, it might be simpler to just replace policing with occasional checkups.

"So, committed any crimes this month?"

Comment author: NancyLebovitz 21 October 2011 06:51:05AM 2 points [-]

Self-perceived truth value sounds like a subtle problem.

I'd settle for memory testing for eye-witnesses, though memory testing under stress is probably too much to ask.

Comment author: Desrtopa 21 October 2011 05:05:57PM 11 points [-]

My high school psychology teacher gave my class an interesting demonstration on the usefulness of eyewitnesses.

For one class, we found a notice on the door saying that the class had been relocated to another room. We went to that room, and shortly after our teacher arrived, followed by another teacher. He complained that she had not gotten proper clearance to move her class to that room, and he needed it for an exercise for his own class. They spent a couple minutes arguing, and harsh words were exchanged, after which he left the room.

Almost immediately after he left, our teacher asked us to create profiles of his physical description. Estimates of his height ranged from 5'6 to 6'3, his hair was variously described as being brown, black, or red, and his weight was somewhere from 140 pounds to 230.

The information that the students retrieved from their short term memory, which had not yet been encoded as long term memory, was already profoundly unreliable.

Comment author: dbaupp 21 October 2011 06:12:25AM 1 point [-]

"So, committed any crimes this month?"

This would probably only work if the person knew activity X was a crime. There are probably ways of becoming convinced that X wasn't a crime (or vice versa).

Comment author: Desrtopa 21 October 2011 06:27:07AM 1 point [-]

This occurred to me, but I suspect that it would still be easier to catch more people this way than with active policing, particularly for the crimes you most care about catching people at, which most people will already know are wrong and probably if not definitely illegal.

And you could always ask them, "Are you cultivating deliberate unawareness?"

Comment author: gwern 21 October 2011 02:14:31AM 0 points [-]

Such a system would be far from most current Western legal systems and hence is of not much interest to me in the context of discovering bounds on error rates in current Western legal systems.

Comment author: satt 20 October 2011 11:16:36PM 0 points [-]

I thought the same thing.

It also occurred to me that (although it's unlikely) we might discover a neural signature distinguishing spurious memories from accurate ones. Or some less powerful but still useful signature, such as one that distinguishes memories that have been accessed after creation (and so potentially overwritten) from memories that have never been accessed (which are presumably more reliable).

Comment author: falenas108 21 October 2011 01:52:57AM 0 points [-]

There is weak evidence that the memories you make during the day are reviewed during REM sleep, which would mean every memory is gone over at least once.

Comment author: Jack 20 October 2011 08:19:30PM 2 points [-]

Even just reliable brain-scan based lie detectors...

Comment author: PhilGoetz 21 October 2011 04:12:12AM *  7 points [-]

I want to see the regression of LW karma (or the log of LW karma) on probability-Amanda-is-guilty!

You ought to leave Eliezer out of the equation, or assign him a Karma value equal to Yvain's, or else he'll dominate the regression.

Comment author: Jack 21 October 2011 06:20:07AM *  12 points [-]
Comment author: MinibearRex 21 October 2011 02:40:13PM 4 points [-]

Doesn't look like there's any sort of function you can use, but there are almost no points in the top right corner of your graph. Almost nobody with high karma on LW assigned a substantial probability that Knox was guilty.

Comment author: PhilGoetz 21 October 2011 10:25:16PM *  1 point [-]

Neat! It appears the shared version is writable. How can I make a copy of a Google doc, so I can mess around with it myself? I entered this into OpenOffice, and it also gave me a slope of -.02, an intercept of .4, and an R-squared of .02. Weird that the R-squared is so low, since 8 people with log(karma) < .7 gave p > .5, and no people with log(karma) > .7 did.

Also, Eliezer gave p=.15, and that doesn't appear on the graph.

Comment author: komponisto 22 October 2011 12:36:08PM 6 points [-]

Also, Eliezer gave p=.15, and that doesn't appear on the graph.

No he didn't.

Comment author: Jack 24 October 2011 06:05:17AM 2 points [-]

Left Eliezer off since we don't have a firm probability for him and his karma is a huge outlier.

Comment author: Jack 21 October 2011 11:12:59PM 1 point [-]

There is a copy option under the file tab. You're welcome to add to mine though.

Weird that the R-squared is so low, since 8 people with log(karma) < .7 gave p > .5, and no people with log(karma) > .7 did.

For users with Ln(karma) > 7 no answer was > 0.5. But within that range people were about as attracted to 0.5 as they were to 0.01. For further investigation I'd want like to see residuals and the log10 of the prediction.

Comment author: dbaupp 21 October 2011 06:15:24AM 2 points [-]

Do you want it enough to do it yourself?

Comment author: bigjeff5 07 November 2011 09:14:07PM *  6 points [-]

I updated my 60% guilt for Knox/Sollecito almost immediately after reading the follow-up article. As I noted on that page, my 60% judgement was a clear case of anchoring. I started with the pro-guilt evidence and only managed an 80% guilt after reading only their evidence. That I was then only able to re-adjust down to 60% was absurd. Since I was (and still am) fairly weak as a rationalist, I probably should have withheld any kind of assessment until after I had read the pro-innocence website, rather than try to make an early assessment and update it with the evidence I knew would be coming.

I didn't put a number on my update at the time, I just went from "There isn't enough evidence to convict" to "Why were they still suspects?". If I had to put a number on it, I'd say this was a <5% chance of guilt.

Since the acquittal, given the judge's statements about the DNA evidence and the handling thereof (I didn't read the full assessment by the independent expert), I'd have to adjust it now to <1%.

ETA: I just went through the independent analysis, and the conclusions were even stronger than I thought they were.

I thought that the conclusions were of the nature: "this DNA cannot be conclusively matched to anyone".

In fact, they were of the nature: "There is no organic material present."

They did find a few grains of starch on the bra clasp, but that was it. How did the Scientific Police screw it up that badly? They got a shakey DNA match where no DNA existed!

I have to update my assessment yet again, to the lowest I'm willing to put a number on, and that's <0.1% chance that either of them are guilty. There is literally zero evidence that either of them were involved.

Comment author: Jack 20 October 2011 04:45:35PM *  6 points [-]

Amazing job putting this together.

I had remembered LW as being a hive of Amanda fans, but either I’m succumbing to hindsight bias or people updated significantly after those articles.

My .13 probability in the first thread definitely went down following further discussion and, in particular, komponisto's second post. In the last year I've been comfortable using "She's definitely innocent" talking about the case with non-LWers.

Comment author: Eliezer_Yudkowsky 20 October 2011 07:43:30PM 13 points [-]

See my added comment. I did not assign a probability of 15%. I said that if you assigned a probability higher than 15%, it meant you had a really major problem with crediting the opinions of other people and the authority of idiots. My probability that Knox and Sollecito were guilty was "that's privileging the hypothesis", i.e., "I see no real evidence in its favor so same as prior probability", i.e., "really damned' low".

Comment author: gwern 20 October 2011 10:47:35PM 2 points [-]

When people gave ranges, I just used the anchoring number. You gave a range starting at 15%, so that's what I listed.

Comment author: Eliezer_Yudkowsky 21 October 2011 03:27:01PM *  8 points [-]

That's a neat compact algorithm but this doesn't change the fact that it produces the wrong answer.

Again, 15% isn't the maximum of a range. It's a number that's not just "wrong" but "sufficiently wrong to imply you need to adjust your emotional makeup".

If you need a number for me, put in "<0.01". I wouldn't have bet $20,000 at 99-to-1 odds over it at the time of writing that first paragraph, but I'm not quite sure anymore that this really means my probability is >0.01, it's not like I'd have taken the bet the other way.

Comment author: APMason 20 October 2011 05:16:22PM *  7 points [-]

Strange thing about this is, if I've calculated it right, the average probability estimate of Guede's guilt is only ~87%. It seems to me that if this were your real probability estimate of his guilt, and you were on the jury at the guy's trial, you would be obligated to vote innocent. If you operate on the basis that a 13% chance of innocence is not a reasonable doubt, about thirteen out of every hundred people who go to jail will be innocent. That is (let me check) more than one in ten, which strikes me as rather a lot. I think my own estimate of Guede's guilt is above 99%, so I would vote guilty, but I'm surprised the average here is so low.

Comment author: steven0461 20 October 2011 06:44:27PM 14 points [-]

If you operate on the basis that a 13% chance of innocence is not a reasonable doubt, about thirteen out of every hundred people who go to jail will be innocent.

That's if everyone who went to jail had a 13% chance of innocence. Presumably much of the time it would be lower.

Comment author: APMason 20 October 2011 07:03:11PM 1 point [-]

Yes indeed. My mistake.

Comment author: byrnema 20 October 2011 11:35:48PM 3 points [-]

Not 13 out of every 100 people in jail, but still 13 out of every 100 people sentenced by the jury as guilty in the case of a probability estimate of only ~87%. ....The argument still works to show that the probability of guilt at 87% is too low to vote guilty.

Comment author: Peter_de_Blanc 21 October 2011 07:04:34AM 2 points [-]

This argument does not show that.

Comment author: byrnema 21 October 2011 11:12:57AM 0 points [-]

Which argument? I meant the argument loosely defined as the one where you count which fraction of innocent people are jailed to determine if the probability of guilt at 87% is appropriate. Steven0641 correctly pointed out that the target space for the fraction isn't all people in jail, but then you modify the target space to all people judged guilty with probability 87% and the argument 'works'.

Comment author: steven0461 21 October 2011 07:43:21PM 0 points [-]

The argument works if adding a 13% innocent population to jail is clearly wrong even though sending an individual with 13% probability of innocence to jail is not clearly wrong. Peter's point, I think, is that we don't have that "if".

Comment author: byrnema 21 October 2011 08:38:06PM 0 points [-]

I would say that sending an individual with 13% probability of innocence to jail is clearly wrong, because 1 out of 10 of them would be innocent.

So the premise instead is: adding a 13% innocent population of any subset or category of individuals to jail is clearly wrong

leading to the conclusion: sending an individual with only 87% probability of guilt to jail is wrong

Comment author: lessdazed 22 October 2011 02:53:21AM 2 points [-]

I would say that sending an individual with 13% probability of innocence to jail is clearly wrong, because 1 out of 10 of them would be innocent.

One wonders how many of those are people the jury correctly thinks have done other crimes, or subjectively think deserve more punishment for past crimes. That would be a different malfunction from the expressed intent of the system and would imply the system otherwise does much better than the 87/13 ratio.

Comment author: steven0461 21 October 2011 08:58:30PM *  1 point [-]

Yes, that's what I meant by what I said. But the problem is that, at least to me, the premise is no more obvious than the conclusion.

Comment author: byrnema 21 October 2011 09:17:35PM 1 point [-]

I see. It's a little more obvious to spell out "more than 1 out of 10 innocent" instead of "only 87% probablity of guilt" but if you see them as immediately equivalent then indeed the argument will do nothing for you.

Comment author: komponisto 21 October 2011 08:00:12PM *  0 points [-]

I thought that "13% innocent population in jail is wrong" was a premise, and "individual with 13% probability of innocence in jail in wrong" was the conclusion.

Which seems perfectly reasonable to me: if you have an 87% certainty threshold for conviction, it means you're willing to tolerate up to 13% of convicts being innocent, an unacceptably high number by my lights.

Comment author: Luke_A_Somers 21 October 2011 08:50:17PM 2 points [-]

It gets worse - the most severe crimes face the strongest pro-conviction biases.

Comment author: komponisto 21 October 2011 09:29:03PM 1 point [-]

...which is of course exactly the opposite of how it should work.

Comment author: steven0461 21 October 2011 08:24:39PM 0 points [-]

So it sounds like you're saying we do have the "if". But are you sure the number is not just unacceptably high because in any realistic example of a 13% innocent population of convicts, many of them would have to have been seen as having substantially greater than 13% chances of innocence? If not for some biasing effect like that, it's hard for me to see why the moral question would suddenly be clear once it was stated in population frequencies rather than in individual probabilities.

Comment author: komponisto 21 October 2011 09:17:57PM 0 points [-]

So it sounds like you're saying we do have the "if".

Actually, no, because the equivalence of the two formulations is obvious to me.

But it might not be for everyone; it's well known that many people find thinking in terms of frequencies more intuitive than thinking in terms of bare probabilities. For such people, a statement about probabilities may simply not have any moral force unless and until it is translated into a statement about frequencies.

Comment author: Nornagest 21 October 2011 08:56:08PM *  0 points [-]

But are you sure the number is not just unacceptably high because [...] many of them would have to have been seen as having substantially greater than 13% chances of innocence? If not for some biasing effect like that, it's hard for me to see why the moral question would suddenly be clear once it was stated in population frequencies rather than in individual probabilities.

Well, people's intuitions about justice aren't all that consistent, so I don't think this particular moral question is going to suddenly become clear to all observers no matter how it's stated. That being said, though, I don't think we have any particular reason to think that Guede was convicted on unusually shaky evidence, so it seems reasonable -- given certain assumptions -- to take our estimates of his case as representative of murder cases in general.

A 13% innocence threshold for each particular case won't give you a 13% innocent prison population (assuming good estimates, which is probably generous in this context), but if we adopt that criterion and Guede's in the middle of the probability distribution for murder defendants, it seems likely that the resulting population-level incidence would still land on the bad side of 8 or 10%. Which doesn't look much better.

Comment author: Nornagest 20 October 2011 05:55:56PM 5 points [-]

Well, I haven't looked at those estimates for a few months, but I'd imagine that a lot of the margin in the outside view of Guede's case comes from uncertainties introduced by the media handling of the case or by an imperfect view of the evidence. Neither of those factors would, presumably, apply to a jury.

That being said, I wouldn't be all that surprised if thirteen out of a hundred prisoners in Europe and the US were innocent of some of the charges that put them in jail. It's higher than my own estimate would be, but within the same order of magnitude.

Comment author: APMason 20 October 2011 06:03:45PM 0 points [-]

I agree that it wouldn't be hugely surprising. I meant it strikes me as higher than acceptable.

Comment author: billswift 20 October 2011 05:59:29PM *  4 points [-]

In a situation like a trial, where I would be limited to just those "facts" presented by the lawyers, it would be extraordinarily unlikely for me to give better than 90% probability of anything.

Comment author: AlexSchell 20 October 2011 07:16:04PM *  1 point [-]

I would think that hypothetical juror judgments of guilt or innocence may be a lot more prone to bias than a more "dispassionate" look at the evidence generating a probabilty estimate. Even if one should count one's own hypothetical guilt/innocence judgment as a small bit of evidence in the right direction, explicitly trying to calibrate this judgment with one's prior probability estimate is going to make one over-correct one's estimate.

Comment author: Psychohistorian 31 October 2011 05:16:54AM 3 points [-]

Since I was randomly chosen to comment on this, I'll throw in my two cents. I haven't thought about too much and my first instinct was to trust whatever value judgements I had made at the time, which I thought were something like 5-5-95, but were actually 1-1-99. Since me-at-the-time was much more familiar than me-right-now, I'd still probably defer to his judgement; if anything, her exoneration and other evidence should move those numbers slightly closer to the extremes.

Comment author: Matt_Simpson 22 October 2011 11:49:35PM 3 points [-]

I assigned a relatively high probability of guilt for Amanda Knox because of a combination of ignorance and over-correction. I read material on the website of the innocent side first, and felt fairly convinced largely because of their assessment of the DNA evidence. The other website had a conflicting assessment of the DNA evidence, which I didn't know how to adjudicate without trying to learn more about how DNA evidence works. I didn't do this, and so remained uncertain but still thought that Knox was innocent with fairly high probability.

Additionally, I recall at the time that I was afraid of being "wrong." The original post had the feel of setting up a "gotcha" and my instinct was to avoid being gotcha'd, so I started to hedge. I realized this wasn't a good reason to change my assessment of the situation, but I discovered something else to blame it on: my own personal biases. I used to be a pretty hardcore libertarian and am wary of politics hijacking my mind (again), so I blamed the hedging on a correction of my own biases - I think I even mentioned this bias in my original comment. I also probably overweighted the fact that Knox was accused in the first place in my assessment.

In any case, I also recall changing my mind after reading the followup article and comments - partially because more research was put into that article than I put into my own assessment, but also partially because they considered the relevant data more honestly than I had. I don't really recall the details of this update or, for that matter, the real reason why I now think Amanda Knox is innocent - it's all cached thoughts at this point.

Comment author: komponisto 22 October 2011 09:02:04PM *  3 points [-]

One person whose reflections would be particularly welcome, of course, is Rolf Nelson.

Our debate (which is currently stalled due to my fault -- it's my turn to reply) has dealt with one of the few important pieces of evidence to emerge after the original verdict: the incompatibility of the digestive evidence with the prosecution's hypothesized time of death. (This was covered during the original trial, but was never a focus of discussion among outside commentators until the folks at the JREF forum brought it to attention last year.)

Comment author: jimmy 22 October 2011 06:53:44PM 3 points [-]

My high estimate came from spending insufficient mental energy to come to a stable estimate as well as, as Eliezer said "if you assigned a probability higher than 15%, it meant you had a really major problem with crediting the opinions of other people and the authority of idiots."

Comment author: brazil84 20 October 2011 07:47:27PM 3 points [-]

I can't remember exactly what I said in the last thread, but I think my opinion is basically the same now. I am reasonably confident that Knox and Sollecito were involved in the murder and very confident that Guede was.

Comment author: TrE 20 October 2011 08:28:12PM 2 points [-]

What evidence did you collect since that last post, and how shifted it your views (rather, how did it fail to shift them)?

Comment author: brazil84 20 October 2011 09:10:05PM 1 point [-]

The main evidence was that the appeals court in Italy reversed the convictions of Knox and Sollecito. This undermined my confidence a bit. On the other hand, arguing that Knox and Sollecito were involved in the murder has made me a bit more confident in my beliefs. I hope that this is because arguing has given me the opportunity to think more carefully about the case, but it may also be the false confidence which comes from emotional investment in a position. Either way, I doubt it's made much of a difference since I was pretty confident from the beginning.

Comment author: Jack 20 October 2011 09:16:15PM 3 points [-]

Did you at any point update on your fellow Less Wrong posters' estimates?

Comment author: brazil84 20 October 2011 09:49:40PM 2 points [-]

I'm not sure what you are asking, but the opinions of other posters here has not had much of an impact on my own.

Comment author: Jack 20 October 2011 10:53:08PM *  3 points [-]

Well, the judgment of people here is evidence just like anything else. Lets say I initially predicted Knox's guilt with p=0.01, Since I think my beliefs track the truth and the beliefs of other Less Wrong posters track the truth I should expect other posters to agree with my assessment if my belief is accurate. The majority of posters disagreeing with me is far more likely if I'm wrong than if I'm right. So upon learning that the vast majority of posters disagree with me I should be more uncertain about my prediction.

How uncertain I should be is a difficult question-- in many cases in that thread it was resolved by discussing evidence. Many people with initially high probabilities shifted their estimates downward after evidence they missed was pointed out to them. If you think you have evidence other Less Wrong posters don't have then it makes sense to not take their opinions seriously. Alternatively, if you think Less Wrong posters are irrational or poorly calibrated and don't expect their beliefs as a group to track the truth well then it makes sense to more or less ignore their opinion. I suppose one could also ignore the opinions of the Less Wrong posters on the ground that the opinions of random people reading about the case are swamped by the opinions of people who have studied the case for months-- and thus make very little difference. But now Knox and Sollecito have been released-- if your trust in the experts was what lead you to ignore Less Wrong you should update on the new court decision.

So why didn't you update on the opinions of Less Wrong posters?

Comment author: MarkusRamikin 21 October 2011 12:57:11PM *  3 points [-]

I wonder. The opinions of members of a given community are not independent events. There's influence by high status members, and by perceived community consensus (note how in a previous post, brazil84 got downvoted just for admitting, when asked, that this consensus didn't move his own opinion much - I don't know, but to me that's ominous). So isn't there's a risk of counting the same evidence (the arguments and facts that convinced the "first movers" in forming this community consensus) multiple times?

What you say, that if others of my group disagree with me and I'm in a strong minority, then I'm probably wrong - how far does that go? The majority of humanity is probably wrong about a lot of things that we on Less Wrong are probably right about, by virtue of our greater rationality, and we don't seem to be updating in their direction, are we? Well, if brazil84 is a lawyer, then similarly, by virtue of his expertise, it seems reasonable to me that he should not easily let his opinion be influenced by that of laymen.

Comment author: komponisto 21 October 2011 01:29:07PM *  5 points [-]

Well, if brazil84 is a lawyer, then similarly, by virtue of his expertise, it seems reasonable to me that he should not easily let his opinion be influenced by that of laymen.

That might make sense if the question under discussion were a legal question (e.g. how a statute is likely to be interpreted by a court). But that isn't the case here. In fact, even if the domain that brazil84 is claiming expertise in -- determining whether people are telling the truth or not -- were one in which lawyers were more likely to have expertise (and frankly I know of no reason to believe this), the fact is that it has precious little relevance to this case. This case is not about which human statements to believe. Instead, it's about applying Occam's Razor to physical evidence.

Comment author: MarkusRamikin 22 October 2011 07:54:31AM 3 points [-]

Point taken.

Comment author: brazil84 21 October 2011 12:39:23AM 1 point [-]

So why didn't you update on the opinions of Less Wrong posters?

It's a combination of having little respect for the opinions of anonymous internet posters as well as faith in my own ability to look at incomplete evidence concerning real world disputes and draw reasonable conclusions. As an attorney I do this every day. In fact, my livelihood depends on doing it. All day long people call me up and spin tales and I have to guess at what happened in their case based on limited evidence. I've been wrong many times over the years, both in believing people who turned out to have been BSing me as well as being skeptical of people who turned out to have been telling the truth.

Comment author: wnoise 21 October 2011 05:38:21PM 11 points [-]

anonymous internet posters

Pseudonymous. There are many similarities, but having a long-standing name does have significant differences, even if the name isn't tied to one's "real-life" name.

Comment author: Prismattic 21 October 2011 01:12:20AM 8 points [-]

There seems to be a certain disjoint between the second half of this paragraph and the first.

Comment author: RomanDavis 21 October 2011 05:00:11AM 0 points [-]

Confidence isn't really about evidence?

Comment author: Desrtopa 21 October 2011 04:48:37AM 5 points [-]

It's a combination of having little respect for the opinions of anonymous internet posters as well as faith in my own ability to look at incomplete evidence concerning real world disputes and draw reasonable conclusions.

Keep in mind that you are yourself an anonymous internet poster dealing with other anonymous internet posters with confidence in their ability to look at incomplete evidence concerning real world disputes and draw reasonable conclusions. I would say this is a situation where consideration of the outside view is warranted.

Comment author: brazil84 21 October 2011 05:13:20AM 2 points [-]

Keep in mind that you are yourself an anonymous internet poster dealing with other anonymous internet posters

Well to me, I'm not anonymous. But anyway, I also try to go by peoples' actual arguments. I think this is a reasonable amount of consideration.

Comment author: Desrtopa 21 October 2011 05:27:05AM *  3 points [-]

Well to me, I'm not anonymous.

Which is a very tenuous basis on which to put yourself in a separate reference class.

You should adjust your confidence according to the strength of others' arguments relative to what you would expect given your prior confidence value, and you should also adjust your confidence according to the fact of others' belief weighted according to your confidence in their mechanisms for establishing truth.

If I believe proposition A, and someone gives me argument X for disbelieving it, and I find argument X weak, I should adjust my confidence little if at all. But if a large population of people whose judgment I have no reason to believe is less sound than my own for cases in this class tells me that proposition A is wrong on the basis of argument X, and I'm just not getting it, I should significantly decrease my confidence, on the likelihood that I really am just not getting it.

Comment author: loqi 21 October 2011 06:09:07AM 8 points [-]

So, to summarize why you didn't update:

  • You didn't know the names of the people commenting.
  • You have faith that you're more reliable than those people.
  • You would lose your job if you weren't so great at seeing through bullshit.
  • You have often failed to see through bullshit.

Boy was Upton Sinclair ever right.

Comment author: brazil84 21 October 2011 10:43:58AM 2 points [-]

•You didn't know the names of the people commenting.

I'm not sure that's the way to put it, but let me ask you this: How much stock do you put in the unsupported assertion of an anonymous person on the internet?

•You have faith that you're more reliable than those people.

Please quote me where I made that assertion.

•You would lose your job if you weren't so great at seeing through bullshit.

Well I need to be decent at a minimum. But basically yeah. I assess cases day in and day out. That's a huge advantage. I know that I'm much better than I was 15 years ago, even though I was just as smart then as I am now.

•You have often failed to see through bullshit.

Sure, getting this kind of feedback is a good way to improve one's judgment. Do you seriously disagree?

Boy was Upton Sinclair ever right.

:shrug: I agree, but employment is sadly not the only motivator for self-deception. Let me ask you this:

Do you agree that the tone of your post is a bit nasty?

Comment author: Desrtopa 21 October 2011 02:46:16PM 3 points [-]

Please quote me where I made that assertion.

To the extent that you don't think that you're more reliable than those people, you're engaging in a treatment of evidence that is simply wrong. The fact of someone's belief is evidence weighted according to the reliability of their mechanisms for establishing belief. That's the principle behind Aumann's Agreement Theorem.

Comment author: Vive-ut-Vivas 21 October 2011 02:25:23PM 6 points [-]

I'm not sure that's the way to put it, but let me ask you this: How much stock do you put in the unsupported assertion of an anonymous person on the internet?

How much stock do you put in the supported assertion of an anonymous person on the internet? I think that's a more relevant question here. To what degree does a poster's anonymity detract from his argument?

Comment author: loqi 21 October 2011 05:17:35PM *  -1 points [-]

Do you agree that the tone of your post is a bit nasty?

Yes. It's a combination of having little respect for the feelings of typically-wrong pseudonymous internet posters as well as faith in my own ability to look at incomplete justifications for sloppy reasoning and draw snarky conclusions.

Comment author: thomblake 24 October 2011 06:27:04PM *  0 points [-]

Nobody seems to have answered this question directly, though it seems easy...

•You have faith that you're more reliable than those people.

Please quote me where I made that assertion.

See the direct parent of the post you were replying to (which I think should have been obvious since it was presented as a summary):

It's a combination of having little respect for the opinions of anonymous internet posters as well as faith in my own ability to look at incomplete evidence concerning real world disputes and draw reasonable conclusions.

Also, don't you at least see the tension between:

You would lose your job if you weren't so great at seeing through bullshit.

You have often failed to see through bullshit.

It seems the logical conclusion is that you've lost your job.

Comment author: wedrifid 21 October 2011 01:33:53AM 9 points [-]

And here we have a case study on what not to do and why.

Comment author: brazil84 21 October 2011 01:39:20AM 0 points [-]

If you want to make an argument for why I should put more weight on other posters' opinions about Knox and Sollecito, I'm happy to consider it.

Comment author: wedrifid 21 October 2011 01:44:55AM *  11 points [-]

I'm afraid this is a lesson for others to learn by observation and not one which you can learn yourself (without changing your mind). The reasoning goes along the lines:

  • brazil84 didn't learn from the opinions and reasoning of other fairly rational and intelligent people.
  • brazil84 expended sufficient energy on the topic in question to be able to arrive at a sane conclusion.
  • brazil84 did not arrive at a sane conclusion.
  • Don't do what brazil84 did because it makes you wrong and also makes you look silly.

Note that this is both an somewhat opposing but also complimentary lesson to the one Eliezer notes.

Comment author: brazil84 21 October 2011 01:54:09AM 0 points [-]

I'm afraid this is a lesson for others to learn by observation and not one which you can learn yourself (without changing your mind).

I vaguely recall that you got pretty annoyed at me a year or so ago when I pointed out a contradiction in your reasoning. I suspect that your anger at me over that incident is informing your commentary.

But anyway, if there really are any lurkers reading this, feel free to look back at the actual arguments I made concerning Knox and draw whatever you conclusion you like. Also pay specific attention to my exchange with wedrifid.

Comment author: Desrtopa 21 October 2011 02:25:56AM 10 points [-]

I vaguely recall that you got pretty annoyed at me a year or so ago when I pointed out a contradiction in your reasoning. I suspect that your anger at me over that incident is informing your commentary.

I've had no interaction with you on this site at all, but I have read your posts on the previous Amanda Knox threads, and while I believe I have a far greater aversion than wedrifid to making statements so likely to antagonize others, I have to say I find your judgment in this case in conjunction with your position as a lawyer downright frightening.

Comment author: magfrump 21 October 2011 04:57:44AM 7 points [-]

I suspect that your anger at me over that incident is informing your commentary.

Wedrifid is just like that. All the time.

Comment author: wedrifid 21 October 2011 02:54:48AM *  4 points [-]

I vaguely recall that you got pretty annoyed at me a year or so ago when I pointed out a contradiction in your reasoning.

I don't recall any conversations with you. (Mind you I expect I would have if I believed you then. Actually being wrong is embarrassing.)

No, from the premise "brazil84 is blatantly and obviously wrong despite paying attention to the topic" "don't do what brazil84 did" is a reasonably good inference to make. But as I noted you don't share that premise so naturally you should not be expected to believe it. This is why you were not the intended audience.

Comment author: pedanterrific 21 October 2011 02:46:36AM 0 points [-]

I vaguely recall that you got pretty annoyed at me a year or so ago when I pointed out a contradiction in your reasoning.

... Also pay specific attention to my exchange with wedrifid.

Are you referring to the 'exchange' that starts around here and continues from there? If so... I'm not so sure bringing this to people's attention is in your best interests.

Comment author: kodos96 20 January 2012 12:02:24AM 2 points [-]

So I'm a few months late to the game here, but I was one of the people selected to give my retrospective on this, so here goes:

My original estimates were .01, .01 and .99. I realize now that my calibration was off in the same way Komponisto has conceded his was: those numbers are way too strong to be rational estimates for something you read about on the internet for an hour or two. If I had it to do over again with benefit of hindsight, I'd probably say something more like .1 .1 .9.

The thing is though, in the time since then I've done quite a bit more reading on the topic, to the point where I now feel much more justifiably confident in my conclusions... so if I had to choose probabilities TODAY, I'd probably pick something pretty similar to my original .01 .01 .99.... ORIGINALLY those numbers were too strong to be justified based on the state of my knowledge at the time, but that's no longer the case.

So basically in the last two years I've come full circle back to my original estimates.

Of the 3 new pieces of evidence you mentioned, #1 is the only one that you might say I updated on. I didn't see any reason to update on the expert report, since I assume it just contains all the same evidence that I'd already updated on.

Comment author: JQuinton 26 October 2011 03:06:09PM 2 points [-]

Hmm... I'm a bit confused about what was supposed to be predicted here. Were we supposed to predict whether Knox would be convicted, or predict whether Knox actually committed a murder? If I had been involved in the original conversations, I would have assigned a very low probability to Knox's actual guilt, but a higher probability to her being found guilty. One is a question that specifically pertains to Knox herself, and the other is a commentary on the state of the Italian justice system.

Comment author: gwern 26 October 2011 05:06:07PM 2 points [-]

predict whether Knox actually committed a murder

Short of a new oracle (like DNA was previously), we will never have any judgement which is reliable enough to convince both sides that the judgement is correct. So, this was not a case of predicting the legal outcome - you will notice I mentioned I got the appeal wrong - but rather a question of how and whether people have changed their probability since then. Did they increase their belief in her guilt? Decrease it? Leave it unchanged due to a complicated convergence of pro and anti evidence? This wasn't an exercise 'you guys made these predictions, they have been vindicated or falsified, please check your new calibration and ponder how to do better in the future' but 'so, what do you guys think now?'

The discussion has been a little more aggressive than I hoped for, but I still see changing opinions, which is healthy: it'd be strange if the passage of time didn't change one's belief at all!

Comment author: thomblake 26 October 2011 05:11:45PM 0 points [-]

Leave it unchanged due to a complicated convergence of pro and anti evidence?

For anyone who actually worked out figures before and after, this seems like it would be the least likely scenario.

Comment author: gwern 26 October 2011 05:28:42PM 0 points [-]

Yes, it is extremely improbable that the evidences would approximately or exactly counterbalance but in fairness I had to mention it, else it wasn't a complete breakdown (greater, less, equal).

Comment author: eridu 24 October 2011 07:14:32PM 1 point [-]

This comment is unrelated to the main article.

I take issue with your Bayes-prayer. I don't mind so much that it seems to be just a normal prayer with some replaced words, rather than being something good in its own right, though I think this would offend other LWers. However, it does violate one message on LW that I've found very important to internalize:

Never confess to me that you are just as flawed as I am unless you can tell me what you plan to do about it.

So, apparently, we're really really bad at bayescraft. What are we going to do about it?

Comment author: Normal_Anomaly 31 October 2011 01:23:00AM *  4 points [-]

I don't mind so much that it seems to be just a normal prayer with some replaced words, rather than being something good in its own right, though I think this would offend other LWers.

It doesn't offend me, and I don't anticipate that it would offend many other LWers. So I'm going to test this with a poll. If you were NOT offended by changing a Christian prayer to be about Bayes, upvote this comment.

EDIT: This isn't actually what eridu was talking about.

Comment author: wedrifid 31 October 2011 01:47:51AM *  3 points [-]

Not offended but I do think it is rather lame. It did make me downvote and mentally dissociate from the post. More "ewww" than "How dare you!".

Comment author: gwern 31 October 2011 03:48:25AM *  0 points [-]

Was it really that lame? Man, now I'm wondering whether the Fate/Stay Night bit of http://lesswrong.com/lw/7z9/1001_predictionbook_nights/ turned off a bunch of readers who simply haven't mentioned it.

Comment author: eridu 10 November 2011 04:01:30PM -1 points [-]

I really liked that, because I think it captured a good part of the essence of tsuyoku naritai. I don't think the Bayes-prayer did.

Comment author: eridu 10 November 2011 03:57:54PM *  1 point [-]

I think this would offend the LW zeitgeist because to me, this seems like awful political art.

I tried to imply that because I don't like linking to the sequences every time I make some point based off of them, but I think you missed that. Taking that one line out of context didn't help.

Edit: To be as explicit as I can, I don't care that this is a christian prayer. I just think it's not well done, because it doesn't reflect tsuyoku naritai.

Comment author: komponisto 10 November 2011 05:00:08PM *  1 point [-]

I think this would offend the LW zeitgeist because to me, this seems like awful political art.

To me it seems humorous. Which is in stark contrast to awful political art such as the poem EY describes in the linked post.

Comment author: eridu 10 November 2011 05:02:24PM 0 points [-]

Why is it humorous besides pointing out that the author is on our side?

Comment author: komponisto 10 November 2011 05:12:09PM *  3 points [-]

It's a combination of irreverence (mocking religion by treating non-divine entities as gods), incongruity (you don't expect a bunch of science-minded techno-nerds to sit around praying), and self-mockery (poking fun at our level of enthusiasm for Bayesian concepts).

Comment author: eridu 10 November 2011 06:39:34PM -1 points [-]

None of that resolves to anything more than "I'm on your team! Go team!"

Comment author: Normal_Anomaly 10 November 2011 11:16:29PM *  0 points [-]

That makes sense. I thought you were saying that other people objected to using language originating with Christianity. Your complaint is perfectly reasonable and I slightly agree with it.

Comment author: eridu 11 November 2011 03:13:15AM -1 points [-]

I apologize for not being more clear.

Comment author: Desrtopa 31 October 2011 04:09:20AM *  1 point [-]

I wasn't offended, but I did find it a bit ridiculous, and not really in a funny way, although in a different context I might have found it funny.

Comment author: Normal_Anomaly 31 October 2011 01:23:21AM 1 point [-]

If you WERE offended by by changing a Christian prayer to be about Bayes, upvote this comment.

Comment author: Normal_Anomaly 31 October 2011 01:23:35AM -3 points [-]

Karma Balance.

Comment author: lessdazed 31 October 2011 01:52:45AM 0 points [-]

These might be relevant.

I mean, these might be rationally Bayesian.

Comment author: gwern 27 February 2012 03:05:37AM 1 point [-]

What are we going to do about it?

If there's anyone here who has made more suggestions than me and tried harder than me on calibrating one's predictions, I would appreciate an introduction so I can pick their brain.

Comment author: eridu 10 April 2012 12:55:55AM 1 point [-]

I'm saying it deserves mention within the prayer, to remind the reciter that it's not enough to confess one's flaws without also forming a plan to obliterate them.

Comment author: lessdazed 31 October 2011 06:56:00AM 1 point [-]

If Bayes is all knowing, that sort of defeats the point, doesn't it?

Comment author: John_Maxwell_IV 27 October 2011 05:42:22AM *  0 points [-]

I'd appreciate being removed as I don't feel like I spent much time reading or thinking before making my estimate and probably never should have commented. (Sorry for ruining your perfectly good and valid experiment...)

Edit: Actually, it doesn't look like I spent all that much less time on a log scale than other folks. It does seem like we didn't really have enough time to get a solid grip on things though.

Comment author: lukeprog 29 October 2011 04:52:47AM -3 points [-]
Comment author: Desrtopa 29 October 2011 05:11:11AM *  2 points [-]

The family of the British girl killed in Italy expressed dismay that the original evidence was unreliable and has accepted that Amanda Knox may be innocent after all.

Is this really true? It isn't reflected in the quotes that I've read so far.

I went and did a bit more reading on the Troy Davis case, and I don't think they're that comparable. In the evidentiary hearing, the court found that several of the alleged recantations by witnesses were not recantations at all; most of the witnesses were still claiming Davis's guilt. The murder weapon was never recovered, so they have no prints.

Considering the unreliability of eyewitnesses, especially when the police are encouraging a particular response, it's a lot less confidence than I'd want before applying the death penalty, setting aside the issue of whether it's ever an appropriate punishment, but I don't think it can reasonably be compared to the Knox and Sollecito trial, where the defendants were almost certainly innocent.

Comment author: komponisto 29 October 2011 06:15:01AM 1 point [-]

The family of the British girl killed in Italy expressed dismay that the original evidence was unreliable and has accepted that Amanda Knox may be innocent after all.

Is this really true? It isn't reflected in the quotes that I've read so far.

Indeed it isn't -- what they have said is that they plan to support the prosecution's appeal of the acquittal. (In fact, as the Italian system allows, they are actually a party to the case, seeking a monetary judgement against Knox and Sollecito. [This fact should not be confused with the recent false reports that they were planning to sue Knox in another jurisdiction.])