= 27d567bc2d48d9f40e9ccc20ea370b15)
brazil84 comments on The $125,000 Summer Singularity Challenge - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (259)
I found Watson to be pretty disappointing.
For one thing, it's big advantage was inhuman button-pushing speed since an actuator is much faster than a human finger. Now, one might argue that pushing the button is part of the game, but to that I would respond that reading the puzzles off of the right screen is part of the game too and Watson didn't have to do that -- the puzzles were inputted in the form of a text file. Also, travelling to Los Angeles is part of the game and Watson didn't have to do that either. If the game had been played in Los Angeles instead of New York, then all of Watson's responses would have been delayed by a few hundredths of a second.
Another problem is that a lot of the puzzles on Jeopardy don't actually require much intelligence to solve particularly if you can write a specialized program for each puzzle category. For example, I would guess a competent computer science grad student could pretty easily write a program that did reasonably well in "state capitals" And of the puzzles which do require some intelligence, the two human champions will split the points.
I'm not saying that Watson wasn't impressive, just that it's win was not convincing.
Watson was not specialized for different categories. It would learn categories -- during a game, after seeing question-answer pairs from it. It ignored category titles, because they couldn't find any way to get that to work. (Hence "Toronto" when the category was "U.S. cities".)
I have a really hard time believing this. A lot of the categories on Jeopardy recur regularly and pose the same types of puzzles again and again. IBM would have been crazy not to take advantage of this regularity. Or at least to pay attention to the category titles in evaluating possible answers.
*shrug* I mean, if you want to claim that the makers of IBM coordinated to lie about this point, go ahead, but don't expect to me to bother discussing this with you at that point.
If your comment was inaccurate, it would probably be because you were mistaken and perhaps something you read was mistaken, not that IBM had coordinated to lie.
Yeah, so as it happens, I was misremembering - it doesn't ignore category titles, it just doesn't weight them very highly. Which FWIW still contradicts what brazil84 was suggesting it does. :P
Here's a quote I found from the IBM research blog:
Seems to me that at a minimum, this shows that Watson does not ignore category titles.
I didn't say it ignores categories -- it knows which questions go together in a category, and learns what to use for a given category as it sees question-answer pairs for it. What I said was that it ignores category titles.
However as it happened I was wrong about this; slight misremembrance, sorry. Watson does note category titles, it just doesn't weight them very highly. Apparently it learned this automatically during its training games. Source: http://www-03.ibm.com/innovation/us/watson/related-content/toronto.html