= 27d567bc2d48d9f40e9ccc20ea370b15)
eli_sennesh comments on Open Thread, May 4 - May 10, 2015 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (215)
Short answer: use differential entropy and differential mutual information.
Differential entropy and Shannon entropy are both instances of a more general concept; Shannon entropy for discrete distributions and differential entropy for absolutely continuous ones.
KL-divergence is actually more about approximations, than about variables dependence; it would be strange to use KL-divergence for your purposes, since it is non-symmetric. Anyway, KL-divergence is tightly connected to mutual information:%20=%20KL(p(x,y)%7C%7Cp(x)p(y)))
Differential mutual information is a measure of variables' dependence; differential entropy is a measure of... what? In the discrete case we have the encoding interpretation, but it breaks in the continuous case. The fact that it can be negative shouldn't bother you, because its interpretation is unclear anyway.
As for (differential) mutual information, it can't be negative, as you can see from the formula above (KL-divergence is non-negative). Nothing weird occurs here.
Brilliant! Thank you so much!