You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Lumifer comments on Open thread, Jul. 11 - Jul. 17, 2016 - Less Wrong Discussion

4 Post author: MrMind 11 July 2016 07:09AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (131)

You are viewing a single comment's thread. Show more comments above.

Comment author: Daniel_Burfoot 12 July 2016 05:39:13PM *  4 points [-]

I have some software I am thinking about packaging up and releasing as open-source, but I'd like to gauge how interesting it is to people other than me.

The software is a highly useable implementation of arithmetic encoding. AE completely handles the problem of encoding, so in order to build a custom compressor for some data set, all you have to do is supply a probability model for the data type(s) you are compressing (I call this "BYOM" - Bring Your Own Model).

One of the key technical difficulties of data compression is that you need to keep the encoder and decoder in exact sync, or the whole procedure goes entirely off the rails. This problem is especially acute for the use case of AE, where you are potentially changing the model in response to every event. My software makes it very easy to guarantee that the sender/receiver are in sync, and at the same time it reduces the amount of code you have to write (basically you don't write a separate encoder and decoder, you just write one class that is used for both, depending on the configuration).

Comment author: Lumifer 12 July 2016 06:09:09PM 0 points [-]

Two basic questions:

(1) What are the immediate practical applications?

(2) How qualified must the user be? (The "all you have to do is supply a probability model" part is worrying :-/)

Comment author: Daniel_Burfoot 13 July 2016 01:02:36PM *  2 points [-]

Basically, if you have your own dataset that you wanted to compress with a special purpose model, you could try doing that. You could try out compression-based tricks for computer vision, like in this paper. You could use it as part of an information theory course if you wanted to show students a real example of compression in practice.

In my view it is quite easy to use, but you still need to be a programmer with some knowledge of stats and information theory.

Comment author: Lumifer 13 July 2016 02:39:15PM 1 point [-]

So it's more of a library and less of an application?

Comment author: Daniel_Burfoot 13 July 2016 04:26:14PM 1 point [-]

Yes.