Open Thread, May 4 - May 10, 2015

Gondolinian

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

In a related matter, if I want to study the information theory of hierarchical Bayesian models, what definition of information should I be reading up on? Shannon entropy seems to only apply to discrete distributions; mutual differential entropy seems like the thing I want, but has "weird" properties like occasionally being negative; K-L divergence seems more like a tool for hypothesizing things about out-of-model error than for examining links between random variables.

What I really want to do is: set up a statistical model, and ask a well-defined, well-informed theoretical question about "How much information do I get about random variable Y if I measure random variable X, where the relationships between their distributions are specified by this model I've got here?" Having asked the theoretical question, I want to maybe derive some maths (an analytical expression/equation for the question I asked would be nice), and then run an empirical experiment where I construct such a model and show how the equation holds numerically.

The ultimate goal is to find out precisely how the "Blessing of Abstraction" mentioned in Probabilistic Models of Cognition is happening, and characterize it as a general feature of hierarchical modelling.

Short answer: use differential entropy and differential mutual information.

Differential entropy and Shannon entropy are both instances of a more general concept; Shannon entropy for discrete distributions and differential entropy for absolutely continuous ones.

KL-divergence is actually more about approximations, than about variables dependence; it would be strange to use KL-divergence for your purposes, since it is non-symmetric. Anyway, KL-divergence is tightly connected to mutual information: $I\(X,Y\$ %20=%20KL(p(x,y)%7C%7Cp(x)p(y)))

Differential mutual informat... (read more)