Expanding further on my previous reply, I believe that the claimed (by Gelman and Shalizi) non-Bayesian nature of model-checking is wrong: the truth is that everything that goes under the name of model-checking works, to the extent that it does, so far as it approximates the underlying Bayesian structure. It is not called Bayesian, because it is not an actual, numerical use of Bayes theorem, and the reason we are not doing that is because we do not know how: in practice we cannot work with universal priors.

So Bayesian ideas are applicable to the problem of model/abstraction error, but we cannot apply them numerically. In fact, that is pretty much what model/abstraction error means -- if we did have numbers, they would be part of the model. Model checking is what we do when we cannot calculate any further with numerical probabilities.

Cf. my analogy here with understanding thermodynamics.

I believe that would be Eliezer's response to Gelman and Shalizi. I would not expect them to be convinced though. Shalizi would probably dismiss the idea as moonshine and absurdity.

ETA: Eliezer on the subject:

So if a mind is arriving at true beliefs, and we assume that the second law of thermodynamics has not been violated, that mind must be doing something at least vaguely Bayesian - at least one process with a sort-of Bayesian structure somewhere - or it couldn't possibly work.

ETA: Why is the grandparent at -4? David Chapman and simplicio may be wrong about this, but neither are saying anything stupid, or so much thrashed out in the past as to not merit further words.