To get a more introductory -- but still quite thorough, and more modern -- Bayesian perspective, I recommend John Kruschke's Doing Bayesian Data Analysis. Ignore the silly cover. The book is engagingly written and informative. As a side benefit, it will also teach you R, a very useful language for statistical computing. Definitely worth learning if you are at all interested in data analysis.
Also, you should learn some classical statistics before getting into Bayesian statistics. Jaynes won't really help with that. Kruschke will help a little, but not much. The freely available OpenIntro Statistics textbook is a very good introduction.
I recommend first reading OpenIntro, then Kruschke, then Jaynes.
Also, you should learn some classical statistics before getting into Bayesian statistics.
I'm of two minds about that. I did classical first, and found it painful. It was just wrong. Recipes without real justification. Jaynes was such a relief after that. He just made sense, step after step.
So I would have wished to have started with Jaynes.
But maybe it's good to learn the horrible way first, so that you really appreciate the right way?
Nah, that seems rather demented. Learn the right way first. Learn Jaynes. He covers the basic classical statistical methods anyway, and in a better fashion than classical statistics classes do. He just makes more sense.
Not too long ago, I asked LessWrong which math topics to learn. Eventually, I want to ask for what the prerequisites for each of those topics are and how I should go about learning them. This is a special case of that.
I'm rereading the sequences and Eliezer seems to love E.T. Jaynes. As part of my rationality self-study, I want to work my way through his Probability Theory: the Logic of Science. What math topics do I already need to understand to prepare myself for this? I learned calculus once upon a time, but not fantastically well, and I plan to start by reviewing that.
Also,
Despite Eliezer's praise of the "thousand-year-old vampire", it there a better book to learn probability theory?
Does anyone want to learn this (or the other math from my post above) with me? I'd love to have a partner or maybe even a work group. Location is no obstacle. [Two caveats: 1. I'm busy with stuff and may not be able to get into this for a few months 2. I hard, but I am incredibly slow at computation (such that on every math test I have ever taken, it took me at least 3 times as long as the second slowest person in the class to finish). You might find that I go to slow for you.]