Regarding frequentism vs. Bayesianity in practical applications, the message I take from Yudkowsky and Jaynes is that frequentists have tended historically to lack apprehension of the fact that their methods are ad-hoc, and in general they fail to use Bayesian power when it is in fact advisable to do so - whereas Bayesians feel they can use ad-hoc approximate methods or accurate methods, whichever is appropriate to the task. This is a case in which a questionable philosophy needn't hamstring someone's thinking in principle, but appears to do so fairly predictably as a matter of fact.

Incidentally I'm surprised that there appears to be so much disagreement about this, given that LW is basically a forum brought into existence on the strength of Yudkowsky's abilities as a thinker, writer and populariser, and he clearly holds frequentism in contempt. It's not necessarily a bad thing that some people here are sympathetic to frequentism - intellectual diversity is good - I'm just surprised that there are so many on a Bayesian rationality forum!

About Maxent: I had in mind chapter 5 of this book by Li and Vitanyi.

We can formulate scientific theories in two steps. First, we formulate a set of possible alternative hypotheses, based on scientific observations or other data. Second, we select one hypothesis as the most likely one. Statistics is the mathematics of how to do this. A relatively recent paradigm in statistical inference was developed by J.J. Rissanen and by C.S. Wallace and his coauthors. The method can be viewed as a computable approximation to the incomputable approach in Section 5.2 [i.e. Solomonoff induction] and was inspired by it. In accordance with Occam’s dictum, it tells us to go for the explanation that compresses the data the most. [...]

This is the MDL (minimum description length) principle.

The ideal MDL principle selects the hypothesis H that minimizes K(H) + K(D|H) [...]

Where K is Kolmogorov complexity.

Unfortunately, the function K is not computable (Section 3.4). For practical applications one must settle for easily computable approximations. [...]

So ideal MDL, like Solomonoff induction, is also incomputable!

They go on to discuss approximations, and on page 390 (I don’t know if you have a copy of the book) they provide a usable approximation to be referred to as “MDL”. Later on page 398 they discuss Maxent, and conclude that that too is an approximation to ideal MDL.

As far as I can see, Maxent is more useful in practical applications than their approximate MDL. I felt that Maxent needed to be defended, since Jaynes considered it to be a major element of Bayesian probability theory; and as far as I can see there is no clearly better practical method of generating priors at this point in time such that Maxent could be considered to be one of Bayesianity’s “legitimate issues” vis a vis frequentism.

See paulfchristiano's examples elsewhere in this thread.

Another example would be support vector machines, which work really well in practice but aren't Bayesian (although it's possible that they are actually Bayesian and I just can't figure out what prior they correspond to).

There are also neural networks, which are sort of Bayesian but (I think?) not really. I'm not actually that familiar with neural nets (or SVMs for that matter) so I could just be wrong.

ETA: It is the case that every non-dominated decision procedure is either a Bayesian procedure or the limit of Bayesian procedures (which I think could alternately be thought of as a Bayesian procedure with a potentially improper prior). So in that sense, for any frequentist procedure that is not Bayesian, there is another procedure that gets higher expected utility in all possible worlds, and is therefore strictly better. The only problem is that this is again an abstract statement about decision procedures, and doesn't take into account the computational difficulty of actually finding the better procedure.