I see at least two other major problems with meritocracy.

First, a meritocracy opens for talented people not only positions of productive economic and intellectual activity, but also positions of rent-seeking. So while it's certainly great that meritocracy in science has given us von Neumann, meritocracy in other areas of life has at the same time given us von Neumanns of rent-seeking, who have taken the practices of rent-seeking to an unprecedented extent and to ever more ingenious, intellectually involved, and emotionally appealing rationalizations. (In particular, this is also true of those areas of science that have been captured by rent-seekers.)

Worse yet, the wealth and status captured by the rent-seekers are, by themselves, the smaller problem here. The really bad problem is that these ingenious rationalizations for rent-seeking, once successfully sold to the intellectual public, become a firmly entrenched part of the respectable public opinion -- and since they are directly entangled with power and status, questioning them becomes a dangerous taboo violation. (And even worse, as it always is with humans, the most successful elite rent-seekers will be those who honestly internalize these beliefs, thus leading to a society headed by a truly delusional elite.) I believe that this is one of the main mechanisms behind our civilization's drift away from reality on numerous issues for the last century or so.

Second, in meritocracy, unless you're at the very top, it's hard to avoid feeling like a failure, since you'll always end up next to people whose greater success clearly reminds you of your inferior merit.