IlyaShpitser comments on Algorithmic Progress in Six Domains - Less Wrong

24 Post author: lukeprog 03 August 2013 02:29AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (30)

You are viewing a single comment's thread. Show more comments above.

Comment author: IlyaShpitser 09 August 2013 05:47:25PM *  0 points [-]

You need to read about universal coding, e.g. start here:

http://en.wikipedia.org/wiki/Universal_code_(data_compression%29

I highly recommend Thomas and Cover's book, a very readable intro on info theory. The point is we don't need to know the distribution from which the bits came from to do very well in the limit. (There are gains to be had in the region before "in the limit," but these gains will track the kinds of gains you get in statistics if you want to move beyond asymptotic theory).