All of Ethan (EJ) Watkins's Comments + Replies

By the law of large numbers, $\frac{1}{N}{\sum }_{i=1}^N\ln Q_{\theta }\left(x_i\right)\to {\sum }_{x}P\left(x\right)\ln Q_{\theta }\left(x\right)$ almost surely. This is the cross entropy of $P$ and $Q_{\theta }$. Also note that if we subtract this from the entropy of $P$, we get $D_{KL}\left(P||Q_{\theta }\right)$. So minimising the cross entropy over $\theta$ is equivalent to maximising $D_{KL}\left(P||Q_{\theta }\right)$.

I think the cross entropy of $P$ and $Q_{\theta }$ is actually $H(P,Q_{\theta })=-{\sum }_{x}P\left(x\right)\ln Q_{\theta }\left(x\right)$ (note the negative sign). The entropy of $P$ is $H\left(P\right)=-{\sum }_{x}P\left(x\right)\ln P\left(x\right)$. Since $D_{KL}\left(P||Q_{\theta }\right)={\sum }_{x}P\left(x\right)\left(\ln \right(P\left(x\right)-\ln Q_{\theta }\left(x\right))={\sum }_{x}P(x\left)\ln P\right(x)-{\sum }^{}$... (read more)

$$D_{KL}\left(P||Q\right)=\sum _x{p}_x(\ln p_x-\ln q_x)=E\left[I_P\right(X)-I_Q(X\left)\right]$$

I think there is a mistake in this equation. $I_P\left(X\right)$ and $I_Q\left(X\right)$ are the wrong way round. It should be:
$$D_{KL}\left(P||Q\right)=\sum _x{p}_x(\ln p_x-\ln q_x)=E\left[I_Q\right(X)-I_P(X\left)\right]$$