It's also not clear to me that Bayesianism is correct in the ideal sense (or even what that means)
Interesting. Do you accept that by Cox's theorems, probability theory is the normative theory of epistemology? Do you accept that a "bayesian" method based on explicitly approximating ideal probability theory will always give a more accurate answer? Do you accept that each of the examples above work because and to the extent that they (nonexplicitly) approximate the correct probability-theory answer (the bayes-structure argument)?
(as for how they do, we can put them in bayesian terms to see. Maximum liklihood methods assume a flat improper prior, and report the mode of the resulting probability distribution. We can immediately see that building in the prior disallows aggregation of different information sources. Only reporting the mode hides confidence interval and goes way off in the presence of skew. Also, we can't apply safety factors sensibly (they involve utility calculation, which involves confidence intervals at the least).)
I don't know much about SVM and PCA, but bayesian logistic regression is easy and superior to max liklihood for most things.
Do you accept that by Cox's theorems, probability theory is the normative theory of epistemology?
Not Cox's theorem, although the complete class theorem is more convincing (as well as dutch book arguments).
Do you accept that a "bayesian" method based on explicitly approximating ideal probability theory will always give a more accurate answer?
Only in the very weak sense that by the complete class theorem there exists a Bayesian method (or a limit of Bayesian methods) that does at least as well as whatever you're doing. So sure, if you really...
Question in title.
This is obviously subjective, but I figure there ought to be some "go-to" paper. Maybe I've even seen it once, but can't find it now and I don't know if there's anything better.
Links to multiple papers with different focus would be welcome. For my current purpose I have a preference for one that aims low and isn't too long.