GuySrinivasan comments on Confidence levels inside and outside an argument - Less Wrong

129 Post author: Yvain 16 December 2010 03:06AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (174)

You are viewing a single comment's thread. Show more comments above.

Comment author: JGWeissman 16 December 2010 06:33:18AM 4 points [-]

It might help to work an example.

Suppose we are interested in an event B with prior probability P(B) = 1/2 which is prior log odds L(B) = 0, and have observed evidence E which is worth 1 bit, so L(B|E) = 1 and P(B|E) = 2/3 ~= .67. But if we are meta uncertain of the strength of evidence E such that we assign probability 1/2 that it is worth 0 bits, and probability 1/2 that it is worth 2 bits, then the expected log odds is EL(B|E) = 1, but the expected probability EP(B|E) = (1/2)*(1/2) + (1/2)*(4/5) = (.5 + .8)/2 = .65, decreasing towards 1/2 from P(B|E) ~= .67.

But what if instead the prior probability was P(B) = 1/5, or L(B) = -2. Then, with the same evidence with the same meta uncertainty, EL(B|E) = L(B|E) = -1, P(B|E) = 1/3 ~= .33, and EP(B|E) = .35, this time increasing towards 1/2.

Note this did not even require meta uncertainty over the prior, only the uncertainty over the total posterior log-odds is important. Also note that even though uncertainty moves the expected probability towards 1/2, it does not move the expected log-odds towards 0.

Comment author: GuySrinivasan 16 December 2010 07:21:51AM 0 points [-]

Oh I see, I thought you were saying something completely different. :D Yes, it looks like keeping the expectation of the evidence constant, the final probability will be closer to 0.5 the larger the variance of the evidence. I thought you were talking about what our priors should be on how much evidence we will tend to receive for propositions in general from things we intuit as one source or something.