The math, or application thereof?

For the latter, Applied Cryptography, by Bruce Schneier is the standard response, and surprisingly readable. I've read other books in the field, but nothing I can think of that's quite as much a "textbook", so this recommendation may or may not officially count.

And of course, a caveat applies to any book on cryptography: don't read it and start coding your own algorithms - anyone can invent a cryptosystem he can't break himself. If you're planning on doing development, the only safe way to handle this stuff is to use well reviewed, well maintained libraries. A textbook will give you a sense of what's available and how things might fit together.