Excellent introduction to information theory! I was a bit concerned about you allowing mutual information to be negative and avoiding the "odds amplification by Bayes factor" version of Bayes's Theorem, but then you went right on and justified those.

To the "further reading" section, I would add the great free textbook on the topic by David MacKay.

Now for some nitpicks: Your informativeness function inf(X) also goes by the standard name surprisal or self-information.

Also, there were more things you could have mentioned for the applications section, like

• how it makes the "off-weight tennis ball problem" much easier by leading you to choose weighings that maximize the expected informativeness (aka entropy). (The problem is this: given 12 tennis balls, one of which is heavier or lighter than the rest, find which one is different, and whether it's heavier or lighter, using a balance scale only three times.)

• how to provide useless answers to people: You don't just give false information, as that has mutual information with the subject matter; you have to visibly make your answers dependent on something random in order to ensure it's not informative.