This is going to sound silly, but...could someone explain frequentist statistics to me?

Here's my current understanding of how it works:

We've got some hypothesis H, whose truth or falsity we'd like to determine. So we go out and gather some evidence E. But now, instead of trying to quantify our degree of belief in H (given E) as a conditional probability estimate using Bayes' Theorem (which would require us to know P(H), P(E|H), and P(E|~H)), what we do is simply calculate P(E|~H) (techniques for doing this being of course the principal concern of statistics texts), and then place H into one of two bins depending on whether P(E|~H) is below some threshold number ("p-value") that somebody decided was "low": if P(E|~H) is below that number, we put H into the "accepted" bin (or, as they say, we reject the null hypothesis ~H); otherwise, we put H into the "not accepted" bin (that is, we fail to reject ~H).

Now, if that is a fair summary, then this big controversy between frequentists and Bayesians must mean that there is a sizable collection of people who think that the above procedure is a better way of obtaining knowledge than performing Bayesian updates. But for the life of me, I can't see how anyone could possibly think that. I mean, not only is the "p-value" threshold arbitrary, not only are we depriving ourselves of valuable information by "accepting" or "not accepting" a hypothesis rather than quantifying our certainty level, but...what about P(E|H)?? (Not to mention P(H).) To me, it seems blatantly obvious that an epistemology (and that's what it is) like the above is a recipe for disaster -- specifically in the form of accumulated errors over time.

I know that statisticians are intelligent people, so this has to be a strawman or something. Or at least, there must be some decent-sounding arguments that I haven't heard -- and surely there are some frequentist contrarians reading this who know what those arguments are. So, in the spirit of Alicorn's "Deontology for Cosequentialists" or ciphergoth's survey of the anti-cryonics position, I'd like to suggest a "Frequentism for Bayesians" post -- or perhaps just a "Frequentism for Dummies", if that's what I'm being here.

You don't want to rely on studies in medical journals because their conclusion-drawing methodologies are haphazard.

I dispute none of this, but so far as I can tell or guess, the main thing powering the superior statistical strength of PatientsLikeMe is the fact that medical researchers have learned to game the system and use complicated ad-hoc frequentist statistics to get whatever answer they want or think they ought to get, and PatientsLikeMe has some standard statistical techniques that they use every time.

Also, I presume, PatientsLikeMe is Bayesian or Bayes-like in that they take all available evidence into account and update incrementally, while every medical experiment is a whole new tiny little frequentist universe.

This is not really an article about PatientsLikeMe being strong, it is an article about the standard statistical methods of academic science being weak and stupid.

Different people might justify vegetarianism by citing the suffering of animals, health benefits, environmental impacts, or purely spiritual concerns. As long as there isn't a camp of vegetarians that claim it does not have e.g. redeeming health benefits, we can more or less add all those opinions together.

I think that this is actually very close to the bible/koran example. If people reach similar conclusions from different reasons, they're probably just rationalizing. It would be very surprising if truly independent aspects of vegetarianism all happen to point the same way.

I guess this means that you and I reach the same conclusion about the bible/koran example, but for different reasons ;-)

ETA: I am more negative about vegetarian evidence than James, but I am also more positive about the theists (cf Unknowns, Michael Vassar). In both cases, I say that they are mistaken about why they hold the beliefs they do, but that doesn't necessarily mean the reason is bad. So maybe my position does not apply to my agreement with James.

Hmm, the AI could have said that if you are the original, then by the time you make the decision it will have already either tortured or not tortured your copies based on its simulation of you, so hitting the reset button won't prevent that.

This kind of extortion also seems like a general problem for FAIs dealing with UFAIs. An FAI can be extorted by threats of torture (of simulations of beings that it cares about), but a paperclip maximizer can't.

Truth value of "global warming has a serious chance of destroying the world" statement is entirely unverifiable to an average person. The media are saying it's true, quoting many leading scientists and politicians. By what mechanism do you suggest people reach an alternative conclusion?

There's outside view, but it's not accepted even here, as many have the same world-destroying beliefs about AI, and countless other subjects.

Hi Carl,

I claimed that Jensen's views are relatively uncontroversial, not that they are entirely so. In making that claim, I wasn't thinking only of Jensen's views about the genetic component of the Black-White gap in IQ scores, but also about his views on the existence of such a gap and on the degree to which such scores measure genuine differences in intellectual ability. Perhaps it was confusing on my part to use Jensen's name to refer to the cluster of views I had in mind. The point I wished to make was that the various views about race and IQ that taw might have had in mind in writing the sentence quoted above are not significantly more controversial today than they were in the past, and are shared by a sizeable portion of the relevant community of experts. As Snyderman and Rothman write (quoted by Gottfredson, p. 54),

On the whole, scholars with any expertise in the area of intelligence and intelligence testing ... share a common view of [what constitute] the most important components of intelligence, and are convinced that [intelligence] can be measured with some degree of accuracy. An overwhelming majority also believe that individual genetic inheritance contributes to variations in IQ within the white community, and a smaller majority express the same view about the black-white and SES [socioeconomic] differences in IQ.

Anecdotally, I myself have become an agnostic about the source of the Black-White differences in IQ, after reading Richard Nisbett's Intelligence and How to Get It.

Or, if the reference class is "science-y Doomsday predictors", then they're almost certainly completely wrong. See Paul Ehrlich (overpopulation), and Matt Simmons (peak oil) for some examples, both treated extremely seriously by mainstream media at time.

I think you are unduly confusing mainstream media with mainstream science. Most people do. Unless they're the actual scientists having their claims deformed, misrepresented, and sensationalised by the media.

This says it all.

When has there been a consensus in the established scientific literature about either certitude of catastrophic overpopulation, or imminent turnaround in oil production?

We have plenty of examples where such science was completely wrong and persisted in being wrong in spite of overwhelming evidence, as with race and IQ, nuclear winter, and pretty much everything in macroeconomics.

Hm. Apparently you also have non-conventional definitions of "overwhelming" and "completely wrong".

The question is how to create a formal decision algorithm that will be able to understand the problem and give the right answer (without failing on other such tests). Of course you can solve it correctly if you are not yet poisoned by too much presumptuous philosophy.