I highly recommend this book, but then it's currently my introduction to both Information Theory and Bayesian Statistics, and I haven't read any others to compare it to. I find it difficult to imagine a better one though.

Clear, logical, rigorous, readable, and lots of well chosen excellent exercises that illuminate the text.