= 27d567bc2d48d9f40e9ccc20ea370b15)
Daniel_Burfoot comments on Case study: abuse of frequentist statistics - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (96)
This is going to sound silly, but...could someone explain frequentist statistics to me?
Here's my current understanding of how it works:
We've got some hypothesis H, whose truth or falsity we'd like to determine. So we go out and gather some evidence E. But now, instead of trying to quantify our degree of belief in H (given E) as a conditional probability estimate using Bayes' Theorem (which would require us to know P(H), P(E|H), and P(E|~H)), what we do is simply calculate P(E|~H) (techniques for doing this being of course the principal concern of statistics texts), and then place H into one of two bins depending on whether P(E|~H) is below some threshold number ("p-value") that somebody decided was "low": if P(E|~H) is below that number, we put H into the "accepted" bin (or, as they say, we reject the null hypothesis ~H); otherwise, we put H into the "not accepted" bin (that is, we fail to reject ~H).
Now, if that is a fair summary, then this big controversy between frequentists and Bayesians must mean that there is a sizable collection of people who think that the above procedure is a better way of obtaining knowledge than performing Bayesian updates. But for the life of me, I can't see how anyone could possibly think that. I mean, not only is the "p-value" threshold arbitrary, not only are we depriving ourselves of valuable information by "accepting" or "not accepting" a hypothesis rather than quantifying our certainty level, but...what about P(E|H)?? (Not to mention P(H).) To me, it seems blatantly obvious that an epistemology (and that's what it is) like the above is a recipe for disaster -- specifically in the form of accumulated errors over time.
I know that statisticians are intelligent people, so this has to be a strawman or something. Or at least, there must be some decent-sounding arguments that I haven't heard -- and surely there are some frequentist contrarians reading this who know what those arguments are. So, in the spirit of Alicorn's "Deontology for Cosequentialists" or ciphergoth's survey of the anti-cryonics position, I'd like to suggest a "Frequentism for Bayesians" post -- or perhaps just a "Frequentism for Dummies", if that's what I'm being here.
The central difficulty of Bayesian statistics is the problem of choosing a prior: where did it come from, how is it justified? How can Bayesians ever make objective scientific statements, if all of their methods require an apparently arbitrary choice for a prior?
Frequentist statistics is the attempt to do probabilistic inference without using a prior. So, for example, the U-test Cyan linked to above makes a statement about whether two data sets could be drawn from the same distribution, without having to assume anything about what the distribution actually is.
That's my understanding, anyway - I would also be happy to see a "Frequentism for Bayesians" post.
Without acknowledging a prior.
Some frequentist techniques are strictly incoherent from a Bayesian point of view. In that case there is no prior.
I believe you and would like to know some examples for future reference.
The OP is one such -- Bayesians aren't permitted to ignore any part of the data except those which leave the likelihood unchanged. One classic example is that in some problems, a confidence interval procedure can return the whole real line. A mildly less pathological example also concerning a wacky confidence interval is here.
Yes; in Bayesian terms, many frequentist testing methods tend to implicitly assume a prior of 50% for the null hypothesis.
A prior gives you as much information as the mean of a distribution. So, can't I by the same token accuse both frequentist and Bayesian statistics of attempting to do probabilistic inference without using a distribution?
I mean, the frequentist uses the U-test to ask whether 2 data sets could be drawn from the same distribution, without assuming what the mean of the distribution is. The Bayesian would use some other test, assuming a prior or perhaps a mean for the distribution, but not assuming a shape for the distribution. And some other, uninvented, and (by the standards of LW) superior statistical methodology would use another test, assuming a mean and a shape for the distribution.
No, not in general, it can give much more or much less; it depends entirely on how detailed you can make your prior. Expanding out e.g. as a series of central moments can give you as detailed a shape as you want. It may reduce to knowing only the mean in certain very special inference problems. In other problems, you may know that the distribution is very definitely Cauchy (EDIT: which doesn't even have a well-defined mean), but not know the parameters, and put some reasonable prior on them -- flat for the center over some range, and approximately using a (1/x) improper prior for the width, perhaps cutting it off at physically relevant length scales.
All that information can be encoded in the prior. The prior covers your probabilities over the space of your hypotheses, not a direct probabilistic encoding of what you think one sample will be.