the obvious question is: why isn't this being used?
These are the moments when I am not sure... the obvious answer is "because people are mostly insane"... but then I feel guilty for thinking about this... but even after reflection this seems like the most likely explanation. (Okay, I could use more polite words and replace "insane" by "having really fucked up epistemic habits". Just to emphasise that the problem isn't in their hardware, it's just the software that is completely crap.)
A random data point: When I studied psychology, when we learned about Freud's ego-defense mechanisms, I was like: wow, this is finally something you could make a "math-style" exam about. (Okay, I know Freud is unpopular here, but please suspend your judgement for the sake of argument.) Because the ego-defense mechanisms have a relatively clear definition. E.g., someone says: "you " and you reply: "no, you and !!!" (without there being any evidence for the other person ); this is called "projection". Or, someone says: "you ", and you reply: "no, actually I " (without any evidence for , and actually some evidence for ), that's called "reaction formation". Etc.
So I was like: This is perfect. Here is how I would make an exam: Write a short story illustrating a use of an ego-defense mechanism, and then ask which of the mechanisms it was. That should be super easy, like elementary-school difficulty, and yet would help to connect the definition with a specific example (which some of my classmates had a problem to do; they just memorized the list of names of the ego-defense mechanisms, without attaching any specific meaning to those words; because memorizing lists was the generally used strategy there). I tried to explain this idea to some classmates, I tried to explain it to a teacher... no success. The typical response was something like: "Yes, such a thing could be done, but what's the point? How is that an improvement over memorizing the list of names? It just takes more time to do." (Except for one girl who studied both psychology and math, and she said: "Yes! Obviously, that's how it should be done.")
So, my conclusion is that some people are so epistemically challenged that they don't even understand the difference between understanding something and not understanding but remembering the passwords. (Did I say "some people"? I meant a huge majority of people, including a huge majority of people with university education, including a huge majority of people who specifically study human thinking and/or education at the university!)
Things like this leave me sad about the mental state of the humankind. (And then I read an alternative explanation, that it's all just a cultish influence of LW making me intolerant towards non-rationalists. No, it's a fucking epistemic abyss. The only thing LW did to me was to point a finger to that abyss and say: "Yeah, we see it, too. You are not halucinating. It really is bad.")
One of many problems with the contemporary university system is that the same institutions that educate students also give them their degrees and grades. This obviously creates massive incentives for grade inflation and lowering of standards. Giving a thorough education requires hard work not only from students but also from the professors. In the absence of an independent body that tests that the students actually have learnt what they are supposed to have learnt, many professors spend as little time as possible at teaching, giving the students light workloads (something most of them of course happily accept). The faculty/student non-aggression pact is an apt term for this.
To see how absurd this system is, imagine that we would have the same system for drivers' licenses: that the driving schools that train prospective drivers also tested them and issued their drivers' licenses. In such a system, people would most probably chose the most lenient schools, leading to a lowering of standards. For fear of such a lowering of standards, prospective drivers are in many countries (I would guess universally but do not know that for sure) tested by government bodies.
Presumably, the main reason for this is that governments really care about the lowering of drivers' standards. Ensuring that all drivers are appropriately educated (i.e. is seen as very important. By contrast, the governments don't care that much about the lowering of academic standards. If they would, they would long ago have replaced a present grading/certification system with one where students are tested by independent bodies, rather than by the universities themselves.
This is all the more absurd given how much politicians in most countries talk about the importance of education. More often than not they talk about education, especially higher education, as a panacea to cure for all ills. However, if we look at the politicians' actions, rather than at their words, it doesn't seem like they actually do think it's quite as important as they say to ensure that the population is well-educated.
Changing the system for certifying students is important not the least in order to facilitate inventions in higher education. The present system discriminates in favour of traditional campus courses, which are both expensive and fail to teach the students as much as they should. I'm not saying that online courses, and other non-standard courses, are necessarily better or more cost-effective, but they should get the chance to prove that they are.
The system is of course hard to change, since there are lots of vested interests that don't want it to change. This is nicely illustrated by the reactions to a small baby-step towards the system that I'm envisioning that OECD is presently trying to take. Financial Times (which has a paywall, unfortunately) reports that OECD are attempting to introduce Pisa-style tests to compare students from higher education institutions around the world. Third year students would be tested on critical thinking, analytical reasoning, problem solving and written communcation. There would also be discipline-specific trials for economics and engineering.
These attempts have, however, not progressed because of resistance from some universities and member countries. OECD says that the resistance often comes from "the most prestigious institutions, because they have very little to win...and a lot to lose". In contrast, "the greatest supporters are the ones that add the greatest value...many of the second-tier institutes are actually a lot better and they're very keen to get on a level playing field."
I figure that if OECD get enough universities on board, they could start implementing the system without the obstructing top universities. They could also allow students from those universities to take the tests independently. If employers started taking these tests seriously, students would have every reason to take them even if their universities haven't joined. Slowly, these presumably more objective tests, or others like them, would become more important at the cost of the universities' inflated grades. People often try to change institutions or systems directly, but sometimes it is more efficient to build alternative systems, show that their useful to the relevant actors, and start out-competing the dominant system (as discussed in these comments).