RichardKennaway comments on Rationality Quotes August 2013 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (733)
David Chapman thinks that using LW-style Bayesianism as a theory of epistemology (as opposed to just probability) lumps together too many types of uncertainty; to wit:
I think he is correct, and LWers are overselling Bayesianism as a solution to too many problems (at the very least, without having shown it to be).
I do not see why any of Chapman's examples cannot be given appropriate distributions and modeled in a Bayesian analysis just like anything else:
Dynamical chaos? Very statistically modelable, in fact, you can't really deal with it at all without statistics, in areas like weather forecasting.
Inaccessibility? Very modelable; just a case of missing data & imputation. (I'm told that handling issues like censoring, truncation, rounding, or intervaling are considered one of the strengths of fully Bayesian methods and a good reason for using stuff like JAGS; in contrast, whenever I've tried to deal with one of those issues using regular maximum-likelihood approaches it has been... painful.)
Time-varying? Well, there's only a huge section of statistics devoted to the topic of time-series and forecasts...
Sensing/measurement error? Trivial, in fact, one of the best cases for statistical adjustment (see psychometrics) and arguably dealing with measurement error is the origin of modern statistics (the first instances of least-squared coming from Gauss and other astronomers dealing with errors in astronomical measurement, and of course Laplace applied Bayesian methods to astronomy as well).
Model/abstraction error? See everything under the heading of 'model checking' and things like model-averaging; local favorite Bayesian statistician Andrew Gelman is very active in this area, no doubt he would be quite surprised to learn that he is misapplying Bayesian methods in that area.
One’s own cognitive/computational limitations? Not just beautifully handled by Bayesian methods + decision theory, but the former is actually offering insight into the former, for example "Burn-in, bias, and the rationality of anchoring".
Agreed about chaos, missing data, time series, and noise, but I think the next is off the mark:
He might be surprised to be described as applying Bayesian methods at all in that area. Model checking, in his view, is an essential part of "Bayesian data analysis", but it is not itself carried out by Bayesian methods. The strictly Bayesian part -- that is, the application of Bayes' theorem -- ends with the computation of the posterior distribution of the model parameters given the priors and the data. Model-checking must (he says) be undertaken by other means because the truth may not be in the support of the prior, a situation in which the strict Bayesian is lost. From "Philosophy and the practice of Bayesian statistics", by Gelman and Shalizi (my emphasis):
...
If anyone's itching to say "what about universal priors?", Gelman and Shalizi say that in practice there is no such thing. The idealised picture of Bayesian practice, in which the prior density is non-zero everywhere, and successive models come into favour or pass out of favour by nothing more than updating from data by Bayes theorem, is, they say, unworkable.
They liken the process to Kuhnian paradigm-shifting:
but find Popperian hypothetico-deductivism a closer fit:
For Gelman and Shalizi, model checking is an essential part of Bayesian practice, not because it is a Bayesian process but because it is a necessarily non-Bayesian supplement to the strictly Bayesian part: Bayesian data analysis cannot proceed by Bayes alone. Bayes proposes; model-checking disposes.
I'm not a statistician and do not wish to take a view on this. But I believe I have accurately stated their view. The paper contains some references to other statisticians who, they says are more in favour of universal Bayesianism, but I have not read them.
Loath as I am to disagree with Gelman & Shalizi, I'm not convinced that the sort of model-checking they advocate such as posterior p-values are fundamentally and in principle non-Bayesian, rather than practical problems. I mostly agree with "Posterior predictive checks can and should be Bayesian: Comment on Gelman and Shalizi,'Philosophy and the practice of Bayesian statistics'", Kruschke 2013 - I don't see why that sort of procedure cannot be subsumed with more flexible and general models in an ensemble approach, and poor fits of particular parametric models found automatically and posterior shifted to more complex but better fitting models. If we fit one model and find that it is a bad model, then the root problem was that we were only looking at one model when we knew that there were many other models but out of laziness or limited computations we discarded them all. You might say that when we do an informal posterior predictive check, what we are doing is a Bayesian model comparison of one or two explicit models with the models generated by a large multi-layer network of sigmoids (specifically <80 billion of them)... If you're running into problems because your model-space is too narrow - expand it! Models should be able to grow (this is a common feature of Bayesian nonparametrics).
This may be hard in practice, but then it's just another example of how we must compromise our ideals because of our limits, not a fundamental limitation on a theory or paradigm.