= 27d567bc2d48d9f40e9ccc20ea370b15)
brian_jaress comments on Case study: abuse of frequentist statistics - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (96)
This is going to sound silly, but...could someone explain frequentist statistics to me?
Here's my current understanding of how it works:
We've got some hypothesis H, whose truth or falsity we'd like to determine. So we go out and gather some evidence E. But now, instead of trying to quantify our degree of belief in H (given E) as a conditional probability estimate using Bayes' Theorem (which would require us to know P(H), P(E|H), and P(E|~H)), what we do is simply calculate P(E|~H) (techniques for doing this being of course the principal concern of statistics texts), and then place H into one of two bins depending on whether P(E|~H) is below some threshold number ("p-value") that somebody decided was "low": if P(E|~H) is below that number, we put H into the "accepted" bin (or, as they say, we reject the null hypothesis ~H); otherwise, we put H into the "not accepted" bin (that is, we fail to reject ~H).
Now, if that is a fair summary, then this big controversy between frequentists and Bayesians must mean that there is a sizable collection of people who think that the above procedure is a better way of obtaining knowledge than performing Bayesian updates. But for the life of me, I can't see how anyone could possibly think that. I mean, not only is the "p-value" threshold arbitrary, not only are we depriving ourselves of valuable information by "accepting" or "not accepting" a hypothesis rather than quantifying our certainty level, but...what about P(E|H)?? (Not to mention P(H).) To me, it seems blatantly obvious that an epistemology (and that's what it is) like the above is a recipe for disaster -- specifically in the form of accumulated errors over time.
I know that statisticians are intelligent people, so this has to be a strawman or something. Or at least, there must be some decent-sounding arguments that I haven't heard -- and surely there are some frequentist contrarians reading this who know what those arguments are. So, in the spirit of Alicorn's "Deontology for Cosequentialists" or ciphergoth's survey of the anti-cryonics position, I'd like to suggest a "Frequentism for Bayesians" post -- or perhaps just a "Frequentism for Dummies", if that's what I'm being here.
I too would like to see a good explanation of frequentist techniques, especially one that also explains their relationships (if any) to Bayesian techniques.
Based on the tiny bit I know of both approaches, I think one appealing feature of frequentist techniques (which may or may not make up for their drawbacks) is that your initial assumptions are easier to dislodge the more wrong they are.
It seems to be the other way around with Bayesian techniques because of a stronger built-in assumption that your assumptions are justified. You can immunize yourself against any particular evidence by having a sufficiently wrong prior.
EDIT: Grammar
But you won't be able to convince other Bayesians who don't share that radically wrong prior. Similarly, there doesn't seem to be something intrinsic to frequentism that keeps you from being persistently wrong. Rather, frequentists are kept in line because, as Cyan said, they have to persuade each other. Fortunately, for Bayesians and frequentists alike, a technique's being persuasive to the community correlates with its being liable to produce less wrong answers.
The ability to get a bad result because of a sufficiently wrong prior is not a flaw in Bayesian statistics; it is a flaw is our ability to perform Bayesian statistics. Humans tend to overestimate their confidence of probabilities with very low or very high values. As such, the proper way to formulate a prior is to imagine hypothetical results that will bring the probability into a manageable range, ask yourself what you would want your posterior to be in such cases, and build your prior from that. These hypothetical results must be constructed and analyzed before the actual result is obtained to eliminate bias. As Tyrrell said, the ability of a wrong prior to result in a bad conclusion is a strength because other Bayesians will be able to see where you went wrong by disputing the prior.
Someone correct me if I'm wrong here, but I don't think even having a strong prior P(H) against the evidence is much help, because that makes your likelihood ratio on the evidence P(E|H)/P(E|~H) that much stronger.
(This issue is one my stumbling blocks in Bayescraft.)
The likelihood ratio P(E|H)/P(E|~H) is entirely independent of the prior P(H)
Or did I misunderstand what you said?
In theory, yes, but we're talking about a purported "unswayable Bayesian". If someone strongly believes leprechauns don't exist (low P(H), where H is "leprechauns exist" ), they should strongly expect not to see evidence of leprechauns (low P(E|~H), where E is direct evidence of leprechauns, like finding one in the forest), which suggests a high likelihood ratio P(E|H)/P(E|~H).
I remember Eliezer Yudkowsky referring to typical conversations that go like:
Non-rationalist: "I don't think there will ever be an artificial general intelligence, because my religion says that can't happen."
EY: "So if I showed you one, that means you'd leave your religion?"
He did mention pulling that off once, but I don't believe he said it was typical.
Thanks, that was what I had in mind.
I'm not entirely sure I understand your point. The example you're citing is more the guy saying "I believe X, and X implies ~Y, therefore ~Y", so Eliezer is saying "So Y implies ~X then?"
But the "X implies ~Y" belief can happen when one has low belief in X or high belief in X.
Or are you saying "the likelihoods assigned led to past interpretation of analogous (lack of) evidence, and that's why the current prior is what it is?
komponisto nailed the intuition I was going from: the likelihood ratio is independent of the prior, but an unswayable Bayesian fixes P(E), forcing extreme priors to have extreme likelihood ratios.
*blinks* I think I'm extra confused. The law of conservation of probability is basically just saying that the change in belief may be large or small, so evidence may be strong or weak in that sense. But that doesn't leave the likelihoods up for grabs, (well, okay, P(E|~H) could depend on how you distribute your belief over the space of hypotheses other than H, but... I'm not sure that was your point)
Okay, point conceded ... that still doesn't generate a result that matches the intuition I had. I need to spend more time on this to figure out what assumptions I'm relying on to claim that "extremely wrong beliefs force quick updates".
P(E) = P(E|H) P(H) + P(E|~H)P(~H)
The quantities P(H), P(E|H), and P(E|~H) are in general independent of each other, in the sense that you can move any one of them without changing the others -- provided you adjust P(E) accordingly.
Thanks, that helps. See how I apply that point in my reply to Psy-Kosh here.