RichardKennaway comments on Rationality Quotes August 2013 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (733)
Agreed about chaos, missing data, time series, and noise, but I think the next is off the mark:
He might be surprised to be described as applying Bayesian methods at all in that area. Model checking, in his view, is an essential part of "Bayesian data analysis", but it is not itself carried out by Bayesian methods. The strictly Bayesian part -- that is, the application of Bayes' theorem -- ends with the computation of the posterior distribution of the model parameters given the priors and the data. Model-checking must (he says) be undertaken by other means because the truth may not be in the support of the prior, a situation in which the strict Bayesian is lost. From "Philosophy and the practice of Bayesian statistics", by Gelman and Shalizi (my emphasis):
...
If anyone's itching to say "what about universal priors?", Gelman and Shalizi say that in practice there is no such thing. The idealised picture of Bayesian practice, in which the prior density is non-zero everywhere, and successive models come into favour or pass out of favour by nothing more than updating from data by Bayes theorem, is, they say, unworkable.
They liken the process to Kuhnian paradigm-shifting:
but find Popperian hypothetico-deductivism a closer fit:
For Gelman and Shalizi, model checking is an essential part of Bayesian practice, not because it is a Bayesian process but because it is a necessarily non-Bayesian supplement to the strictly Bayesian part: Bayesian data analysis cannot proceed by Bayes alone. Bayes proposes; model-checking disposes.
I'm not a statistician and do not wish to take a view on this. But I believe I have accurately stated their view. The paper contains some references to other statisticians who, they says are more in favour of universal Bayesianism, but I have not read them.
Loath as I am to disagree with Gelman & Shalizi, I'm not convinced that the sort of model-checking they advocate such as posterior p-values are fundamentally and in principle non-Bayesian, rather than practical problems. I mostly agree with "Posterior predictive checks can and should be Bayesian: Comment on Gelman and Shalizi,'Philosophy and the practice of Bayesian statistics'", Kruschke 2013 - I don't see why that sort of procedure cannot be subsumed with more flexible and general models in an ensemble approach, and poor fits of particular parametric models found automatically and posterior shifted to more complex but better fitting models. If we fit one model and find that it is a bad model, then the root problem was that we were only looking at one model when we knew that there were many other models but out of laziness or limited computations we discarded them all. You might say that when we do an informal posterior predictive check, what we are doing is a Bayesian model comparison of one or two explicit models with the models generated by a large multi-layer network of sigmoids (specifically <80 billion of them)... If you're running into problems because your model-space is too narrow - expand it! Models should be able to grow (this is a common feature of Bayesian nonparametrics).
This may be hard in practice, but then it's just another example of how we must compromise our ideals because of our limits, not a fundamental limitation on a theory or paradigm.