You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Irgy comments on A probability question - Less Wrong Discussion

6 Post author: PhilGoetz 19 October 2012 10:34PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (27)

You are viewing a single comment's thread. Show more comments above.

Comment author: Irgy 25 October 2012 09:43:38PM *  0 points [-]

Yes, but the key point I was trying to make is that using different features with low correlations does not at all ensure that adding the evidence is correct. What matters is not correlations between the features, but correlations between the experts. Correlated features will of course mean correlated experts, but the converse is not true. The features don't have to be correlated for the experts to make mistakes on the same inputs. It's often the case that they do simply because some inputs are fundamentally more difficult than others, in ways that affect all of the features.

If you've observed that there's low correlations between the experts, then you've effectively already followed my main suggestion: " I would strongly advise you simply make some observations of exactly how the probabilities of the experts correlate". If you've only observed low correlations between features then I'd say it's quite likely you're going to generate overconfident results.

PS Much as I don't like "appeal to authority", I do think it's worth pointing out that I deal with exactly this problem at work, so I'm not just talking out of my behind here. Obviously it's hard to know how well experience in my field correlates with yours without knowing what your field is, but I'd expect these issues to be general.