In information theory, there's the concept of the surprisal, which is the logarithm of the inverse of the expected probability of an event. The lower the probability, the higher the surprise(al). The higher the surprisal, the greater the information content.
(Intuitively, the less likely something is, the more you change your beliefs upon learning it.)
So, yeah, it's pretty enshrined in information theory. Entropy is equivalent to the (oxymoronic) "expected surprisal". That is, given a discrete probability distribution over events, the probability-weighted average surprisal is the entropy.
Incidentally, as part of a project to convert all of the laws of physics into information-theoretic form, I realized that the elastic energy of a deformable body tells you its probability of being in that state, and (by the above argument), it's information content. That means you can explain failure modes in terms of the component being forced to store more information than it's capable of.
Well, it's interesting to me.
You seem like as good a person to ask this as any: Is there a good introduction to information theory out there? How would one start digging into the field?
A monthly thread for posting rationality-related quotes you've seen recently (or had stored in your quotesfile for ages).