I asked a professor about this. She's works at the University of Chicago, in philosophy, but she's friends with a math professor she met as a grad student at Berkeley. Here's what she said, so far as I remember it:

The selection process for math programs, at least at Berkeley, was a little funny. It wasn't so hard to get into the program, but there was a very serious filter, a set of exams, about half way through that weeded about 50% of people out. If you passed that, you could go on to get the PhD, but even then, you really need to prove something to finish. There's a lot of low-hanging fruit though, so you don't have to be super-genius or anything.

Once you get onto the job market, you are filtered for a lot of non-math skills. These are skills having to do with the upkeep of the profession: teaching, being a conscientious faculty member, etc. You're also expected to write well and clearly.

I asked if this caused math talent to go to waste:

It might. Empirically, its very hard to say, since undeveloped talent is probably invisible. But remember that good math departments get hundreds and hundreds of applications every year for no more than a dozen positions. Most of the time, they don't have to make trade offs: they can just go for people who are talented and good writers. There just isn't money to fund all the strictly mathematically talented people, especially if they need the support of writers. Also, I doubt the really off the chart brilliant people ever get excluded for anything like writing ability, though they tend to be good writers too.

So what I took away from this was 1) I was wrong in thinking that math departments don't care about math-extrinsic skills. 2) I was wrong to think these don't filter people out. It hadn't occurred to me that there is more mathematical talent than there is money to develop it. It seems like the problem with academia is kind of just a lack of funding.

EDIT: I might as well add that, needless to say, writing ability was considered important to philosophy too, and a filter at every level, but that's not surprising. She didn't have anything to tell me about physics.

can't bear to write badly and don't have the talent to write well

Wow. This is how I feel about my own writing, expressed more clearly than I could myself. I take ages to write a single sentence because none of the phrasings my brain suggests sound like the kind of thing that I'd want to read.

I think I should ask for empirical input at this point: is it your experience that good mathematicians or scientists are filtered out of academic advancement and access to research money and materials as a result of a poor showing in skills extrinsic to their field? By 'extrinsic' I mean skills that are neither necessary nor sufficient to do mathematical or scientific work well.

Well, it's needless only if bad writing turns out to actually not interfere with their ability to do their jobs well... e.g., if their job doesn't involve communicating clearly, or if it does but the way their writing is bad doesn't interfere with clear communication.