To be quite honest, I only really started to study it after reading Eliezer Yudkowsky's Engines of Cognition, which connected it to what I know about thermodynamics. ( Two blog posts inspired by it.) So, like you, I'm an autodidact on the topic.

Most people would recommend David MacKay's downloadable book, which is written in a friendly, accessible tone. That helped a lot, but I also found it hard to follow at times. That may be due to not having a physical copy though. And it can't be beat as a technical reference or in terms of depth.

Personally, my path to learning about it was to basically read the Wikipedia articles on Information Theory and Kullback-Leibler divergence, and every relevant, interesting link that branches off from those (on or off wikipedia).

ETA: Oh, and learning about statistical mechanics, especially the canonical ensemble was a big help for me too, esp. given the relation to the E. T. Jaynes articles on the maximum entropy formalism. But YMMV.