= 27d567bc2d48d9f40e9ccc20ea370b15)
Psy-Kosh comments on IBM's "Watson" program to compete against "Jeopardy" champions tonight - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (26)
The confidences are supposed to be probabilities? But they often summed to > 100%
Or is it "the procedure for generating the confidences is such that it'll be well calibrated for the highest ranking answer"?
No, sorry, that should say confidences everywhere, not probabilities. I had written it out incorrectly and then edited it, but I missed that one. Fixed now.
What I meant was "for the top three answers, the confidences would sometimes sum to > 100, so how does that work?"
Is the procedure defined as well calibrated only for the top answer, or is there something I'm missing?
The confidence level compares the answer to other answers Watson's given in the past, based on how much the answer is supported by the evidence Watson has and uses. All the answers are generated and scored in parallel. It's not a comparison among the answers generated for a specific question, so it shouldn't necessarily add up to 100.
Quote from Chris Welty at last night's panel: "When [Watson] says 'this is my answer, 50% sure,' half the time he's right about that, and half the time he's wrong. When he says 80%, 20% of the time he's wrong."
Ah, thanks.