Daniel_Burfoot comments on Significance of Compression Rate Method - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (60)
Evaluation of AI research is important.
However, I doubt you can even figure out how to use a human-level explanation of the contents of an image to improve lossless image compression rate, so I'd suggest a different route.
There are already many small competitions in AI research - someone creates a dataset/task with a method of evaluating accuracy (usually agreement with human annotation), and others try to do better.
I'd like to see the use of open/common training and test sets eventually lead to a requirement that systems also be published and reproducible by others, or else the research papers about them are rejected. At that point, you can try whatever compression rate method you like on everybody's systems. I'd work toward that, rather than selling CRM.
CRM as you describe it here, if used to decide funding for AI research competitively, would encourage a lot of work in general data compression and not very much AI, at least for a few years.
This is the key assumption. I don't believe it will work, at first, on arbitrary images. But I think it will work on e.g. the roadside video data, where the variation is due primarily to cars.