how much information (in bits) we gain by finding out the exact value of X given some prior beliefs described by p(x).
Just to clarify, this is the expected/average information, right?
If we observe X = x for some fixed x, we get exactly -log_2 p(x) bits of information, and entropy is the average of this information gain taken over the distribution p(x). For example, suppose X ~ Bern(0.5). Then we have a 50% chance of getting a 0, and thus gaining -log2(0.5) = 1 bit of information, and a 50% chance of getting a 1, and thus gaining -log2(0.5) = 1 bits of information, meaning we will necessarily gain 1 bit of information upon finding the... (read more)
Just to clarify, this is the expected/average information, right?
If we observe X = x for some fixed x, we get exactly -log_2 p(x) bits of information, and entropy is the average of this information gain taken over the distribution p(x). For example, suppose X ~ Bern(0.5). Then we have a 50% chance of getting a 0, and thus gaining -log2(0.5) = 1 bit of information, and a 50% chance of getting a 1, and thus gaining -log2(0.5) = 1 bits of information, meaning we will necessarily gain 1 bit of information upon finding the... (read more)