I think the simplicity on this side of complexity is naive theories that "just make sense" and the simplicity on the other side of complexity is mathematical elegance.

When one of the commenters in the Amanda Knox thread said yesterday that the probability has to be either 0 or 1 because either she did it or she didn't, that sounds simple. The mathematics of Bayesian probability are also simple, in that they can be derived from a few premises and explain a wide variety of disparate situations. But they're not the same sort of simplicity.