It's not a good litmus test until you also point to what you consider the best honest skeptical response - albeit this is often damned hard to do with poor skepticism, cryonics being exhibit A in point.

Thanks, that's a good point. There's some authorial sleight of hand going on with that anecdote: I'm telling it to give the reader the feeling of what it's like to see a smart person fail at something basic because they fail to cross domains, but when writing I couldn't actually come up with a real example that was simple enough to fit in one paragraph.

The kind of real examples I had in mind involve the "tests" that people come up with when trying to diagnose a bug or other kind of breakdown, and they make a basic category mistake like trying to "fix" a keyboard stuck in AZERTY instead of QWERTY by unplugging the keyboard and plugging it back in. (And here again, I'm resorting to a simplified example to get my point across.) They know the hardware/software distinction, but they're failing to apply it to their current situation, and instead falling back on "trying" random things. With some justification, because quite often it's what they see "experts" do...

Thanks for the pointer to the original paper.

I'm not seeing why what you call "the real WTF" is evidence of a problem with frequentist statistics.

Check out the title: abuse of frequentist statistics. Yes, at the end, I argue from a Bayesian perspective, but you don't have to be a Bayesian to see the structural problems with frequentist statistics as currently taught to and practiced by working scientists.

I would hope that any competent statistician, frequentist or not, would be sceptical of a nonparametric comparison of means for samples of size 3!

Me too. But not all papers with shoddy statistics are sent to statisticians for review. Experimental biologists in particular have a reputation for math-phobia. (Does the fact that when I saw the sample size the word "underpowered" instantly jumped into my head count as evidence that I am competent?)

This is going to sound silly, but...could someone explain frequentist statistics to me?

Here's my current understanding of how it works:

We've got some hypothesis H, whose truth or falsity we'd like to determine. So we go out and gather some evidence E. But now, instead of trying to quantify our degree of belief in H (given E) as a conditional probability estimate using Bayes' Theorem (which would require us to know P(H), P(E|H), and P(E|~H)), what we do is simply calculate P(E|~H) (techniques for doing this being of course the principal concern of statistics texts), and then place H into one of two bins depending on whether P(E|~H) is below some threshold number ("p-value") that somebody decided was "low": if P(E|~H) is below that number, we put H into the "accepted" bin (or, as they say, we reject the null hypothesis ~H); otherwise, we put H into the "not accepted" bin (that is, we fail to reject ~H).

Now, if that is a fair summary, then this big controversy between frequentists and Bayesians must mean that there is a sizable collection of people who think that the above procedure is a better way of obtaining knowledge than performing Bayesian updates. But for the life of me, I can't see how anyone could possibly think that. I mean, not only is the "p-value" threshold arbitrary, not only are we depriving ourselves of valuable information by "accepting" or "not accepting" a hypothesis rather than quantifying our certainty level, but...what about P(E|H)?? (Not to mention P(H).) To me, it seems blatantly obvious that an epistemology (and that's what it is) like the above is a recipe for disaster -- specifically in the form of accumulated errors over time.

I know that statisticians are intelligent people, so this has to be a strawman or something. Or at least, there must be some decent-sounding arguments that I haven't heard -- and surely there are some frequentist contrarians reading this who know what those arguments are. So, in the spirit of Alicorn's "Deontology for Cosequentialists" or ciphergoth's survey of the anti-cryonics position, I'd like to suggest a "Frequentism for Bayesians" post -- or perhaps just a "Frequentism for Dummies", if that's what I'm being here.

I'd rather give a lot of the money to GiveWell, earmarked for international charities. They can then decide how much would be effective in the hands of Village Reach.

With what staff? Maybe GiveWell or someone could answer this but right now we have no reason to think they could scale up to a budget 100x what they had last year. We also don't know what level of efficiency they could maintain with the increased size.

GiveWell says they could absorb $2.5 million over the next year. That means if Kevin's math is right 15% of one month's revenue would top them off.

I think this is a very strange situation. There's a billion dollar bill in the middle of the road, and it's been there for four years. Yes, we should figure out how we can pick it up, and use it to purchase utilons, but I think as a sub-question, we really really need to figure out why it hasn't been picked up. Because what Kevin is proposing sounds easy. I spend time on Facebook, and I've been invited to, and often joined, a ridiculous number of advocacy groups -- one million strong for Darfur, support breast cancer awareness, etc. I do not understand how a group hasn't already been started to claim this money for some popular cause, and I don't think we're going to be able to claim it ourselves unless we can understand that.

What is it about us, the public, and what is it about conformity itself that causes us all to require it of our neighbors and of our artists and then, with consummate fickleness, to forget those who fall into line and eternally celebrate those who do not?

-- Ben Shahn, "The Shape of Content"

A "friend" of mine was a fan of using this to argue for Christianity. The idea of never changing one's mind doesn't seem very rational.