[link] Baidu cheats in an AI contest in order to gain a 0.24% advantage

Wei Dai

Some of you may already have seen this story, since it's several days old, but MIT Technology Review seems to have the best explanation of what happened: Why and How Baidu Cheated an Artificial Intelligence Test

Such is the success of deep learning on this particular test that even a small advantage could make a difference. Baidu had reported it achieved an error rate of only 4.58 percent, beating the previous best of 4.82 percent, reported by Google in March. In fact, some experts have noted that the small margins of victory in the race to get better on this particular test make it increasingly meaningless. That Baidu and others continue to trumpet their results all the same - and may even be willing to break the rules - suggest that being the best at machine learning matters to them very much indeed.

(In case you didn't know, Baidu is the largest search engine in China, with a market cap of $72B, compared to Google's $370B.)

The problem I see here is that the mainstream AI / machine learning community measures progress mainly by this kind of contest. Researchers are incentivized to use whatever method they can find or invent to gain a few tenths of a percent in some contest, which allows them to claim progress at an AI task and publish a paper. Even as the AI safety / control / Friendliness field gets more attention and funding, it seems easy to foresee a future where mainstream AI researchers continue to ignore such work because it does not contribute to the tenths of a percent that they are seeking but instead can only hinder their efforts. What can be done to change this?

Such is the success of deep learning on this particular test that even a small advantage could make a difference. Baidu had reported it achieved an error rate of only 4.58 percent, beating the previous best of 4.82 percent, reported by Google in March. In fact, some experts have noted that the small margins of victory in the race to get better on this particular test make it increasingly meaningless. That Baidu and others continue to trumpet their results all the same - and may even be willing to break the rules - suggest that being the best at machine learning matters to them very much indeed.

(In case you didn't know, Baidu is the largest search engine in China, with a market cap of $72B, compared to Google's $370B.)

But pure AI startups are a subset of them and have probably grown along the same trend.

Self-assigned labeling and marketing info seems highly doubtful, given things like the AI Winter and the spread of AI techniques outside of whatever is deemed 'AI' at that moment in time. (Consider Symbolics selling Lisp machines: a hardware platform whose selling points were GC support, large amounts of RAM, high-end color displays, and a rich interpreted software ecosystem with excellent hypertext documentation and networking support, all in a compact physical package, sold for developers and catering to areas like oil field exploration. If a Lisp machine were sold now, would we call its manufacturer an AI company? Or would we call it a Chromebook?) And now that AI is hot, every company which uses some random machine-learning technique like random forests is tempted to brand itself as an AI startup. Trends show as much what is trendy as anything.

All the big tech companies have created big machine learning teams and hired tons of researchers...Peter Norvig commented that "his company [Google] already employed 'less than 50 percent but certainly more than 5 percent' of the world’s leading experts in machine learning".

I am sure they have, but that's very different from the claim being made. If this year you hire, say, a full 50% of a field's researchers, that's still not spending more than that field's cumulative expenses for half a century; that's spending, well, half its expenses that year.

20

[link] Baidu cheats in an AI contest in order to gain a 0.24% advantage

20

20

20

[link] Baidu cheats in an AI contest in order to gain a 0.24% advantage

20

20