Very interesting article! It seemed very surprising to me that the information entropy can be interpreted as the (minimum average) number of yes/no-questions needed to fully determine the state of a system. Especially since the value of the entropy depends on the units (i.e. what logarithm base you use). Is the use of base two related to the fact that the answers can be either 'yes' or 'no' (binary)?
Another question. Suppose you have a system X in 4 possible states {X1,X2,X3,X4} with probabilities {3/8,2/8,2/8,1/8} respectively. The information entropy is ... (read more)
EDIT: Welcome to LessWrong. You may want introduce yourself in the welcome thread
Especially since the value of the entropy depends on the units (i.e. what logarithm base you use). Is the use of base two related to the fact that the answers can be either 'yes' or 'no' (binary)?
Precisely.
{3/8,2/8,2/8,1/8} ... 1.91 bits ... 2 questions.
Am I not looking hard enough?
No, there really aren't. A simpler example is just taking two options, with unequal probabilities (take {1/8, 7/8} for concreteness). Again, you have to ask one question even though the... (read more)
Very interesting article! It seemed very surprising to me that the information entropy can be interpreted as the (minimum average) number of yes/no-questions needed to fully determine the state of a system. Especially since the value of the entropy depends on the units (i.e. what logarithm base you use). Is the use of base two related to the fact that the answers can be either 'yes' or 'no' (binary)?
Another question. Suppose you have a system X in 4 possible states {X1,X2,X3,X4} with probabilities {3/8,2/8,2/8,1/8} respectively. The information entropy is ... (read more)
EDIT: Welcome to LessWrong. You may want introduce yourself in the welcome thread
Precisely.
No, there really aren't. A simpler example is just taking two options, with unequal probabilities (take {1/8, 7/8} for concreteness). Again, you have to ask one question even though the... (read more)