Lumifer comments on Open thread, Jul. 11 - Jul. 17, 2016 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (131)
I have some software I am thinking about packaging up and releasing as open-source, but I'd like to gauge how interesting it is to people other than me.
The software is a highly useable implementation of arithmetic encoding. AE completely handles the problem of encoding, so in order to build a custom compressor for some data set, all you have to do is supply a probability model for the data type(s) you are compressing (I call this "BYOM" - Bring Your Own Model).
One of the key technical difficulties of data compression is that you need to keep the encoder and decoder in exact sync, or the whole procedure goes entirely off the rails. This problem is especially acute for the use case of AE, where you are potentially changing the model in response to every event. My software makes it very easy to guarantee that the sender/receiver are in sync, and at the same time it reduces the amount of code you have to write (basically you don't write a separate encoder and decoder, you just write one class that is used for both, depending on the configuration).
Two basic questions:
(1) What are the immediate practical applications?
(2) How qualified must the user be? (The "all you have to do is supply a probability model" part is worrying :-/)
Basically, if you have your own dataset that you wanted to compress with a special purpose model, you could try doing that. You could try out compression-based tricks for computer vision, like in this paper. You could use it as part of an information theory course if you wanted to show students a real example of compression in practice.
In my view it is quite easy to use, but you still need to be a programmer with some knowledge of stats and information theory.
So it's more of a library and less of an application?
Yes.