= 27d567bc2d48d9f40e9ccc20ea370b15)
PhilGoetz comments on A probability question - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (27)
Wait. This doesn't work. If the prior is 1:2, and you have n experts also giving estimates of 1:2, you should end up with the answer 1:2. None of the experts are providing information; they're just throwing their hands up and spitting back the prior. Yet this approach multiplies all their odds ratios together, so that the answer changes with n.
ADDED: Oh, wait, sorry. You're saying take the odds ratio they output, factor out the prior for each expert, multiply, and then factor the prior back in. Expanded,
(8/2)/(4/6) * (7/3)/(4/6) * (4/6) = 8064/576, 1 / (1+1/(8064/576)) = .933
Great!
You don't have to factor out the priors both times, since putting it back in will cancel one of them out. This is equivalent to considering how one of them will update based on the information the other has.
(8/2)/(4/6) * (7/3) = 336/24, 336/(336+24) = 0.933
This is correct in the special case where all the information that your experts are basing their conclusions off of is independent. Slightly modifying your example, if the prior is 1:2 and you have n experts giving odds of 1:1, they each have a likelihood ratio of 2:1, so you get 1:2 * (2:1)^n = 2^(n-1):1. However, if they've all updated based on looking at the results from the same experiment, you're double-counting the evidence; intuitively, you actually want to assign an odds ratio of 1:1.
The right thing to do here is to calculate for each subset of the experts what information they share, but you probably don't have that information and so you'd have to estimate it, which I'd have to think a lot about in order to do well. Hopefully, the assumption of independence is approximately true in your data and you can just go with the naive method.