I asked a professor about this. She's works at the University of Chicago, in philosophy, but she's friends with a math professor she met as a grad student at Berkeley. Here's what she said, so far as I remember it:

The selection process for math programs, at least at Berkeley, was a little funny. It wasn't so hard to get into the program, but there was a very serious filter, a set of exams, about half way through that weeded about 50% of people out. If you passed that, you could go on to get the PhD, but even then, you really need to prove something to finish. There's a lot of low-hanging fruit though, so you don't have to be super-genius or anything.

Once you get onto the job market, you are filtered for a lot of non-math skills. These are skills having to do with the upkeep of the profession: teaching, being a conscientious faculty member, etc. You're also expected to write well and clearly.

I asked if this caused math talent to go to waste:

It might. Empirically, its very hard to say, since undeveloped talent is probably invisible. But remember that good math departments get hundreds and hundreds of applications every year for no more than a dozen positions. Most of the time, they don't have to make trade offs: they can just go for people who are talented and good writers. There just isn't money to fund all the strictly mathematically talented people, especially if they need the support of writers. Also, I doubt the really off the chart brilliant people ever get excluded for anything like writing ability, though they tend to be good writers too.

So what I took away from this was 1) I was wrong in thinking that math departments don't care about math-extrinsic skills. 2) I was wrong to think these don't filter people out. It hadn't occurred to me that there is more mathematical talent than there is money to develop it. It seems like the problem with academia is kind of just a lack of funding.

EDIT: I might as well add that, needless to say, writing ability was considered important to philosophy too, and a filter at every level, but that's not surprising. She didn't have anything to tell me about physics.