You yourself provided an answer to this. It could be a natural progression from a caste system like that. I disagree that these slaves are treated worse than most civilizations through history have kept their slaves. While in the real world slaves had all sorts of horrible things happen to them, here the only bad thing that happens to them is that they don't get to use magic. Throughout human history, most people lived perfectly fine and happily without magic. While yes, there would be the aspect of having to live in a society where everyone else gets to use magic and you don't, I don't think that just that means that this system gives the slaves "rather worse treatment than most civilizations have allowed with respect to their slaves".

I doubt that "the only bad thing that happens to them is that they don't get to use magic." After all, these are people who're regularly mind-controlled into sacrificing their power against their will. They're kept from using magic because they're not trusted not to be enemies of the system. Do you seriously expect that they'll be well treated aside from the fact that they have no legal right to their own mental autonomy?

This sort of system could arise from a preexisting caste system with a sufficiently low caste already available, but if you're positing something as a historical inevitability, then you can't just handwave something like that away; it's not as if this is something you could get in just any civilization. The Paraiah class itself isn't nearly large enough to account for the proportion of the population we're already discussing.

If you try to expand the sector of the population that's sufficiently low on the totem pole as to receive no right to mental autonomy, then you could be looking at large scale class revolts before you have a chance to implement the vows on more than a small sector of the population.

I think that you are comparing this to the wrong things. This is not just a better education system or a public works system, this is the absolute removal of a set problems that has plagued humanity since there was such a thing as humanity.

I would prefer to offer the analogy as something like this: This (Unbreakable Vow based) system is to the current system as democracy is to a dictatorship (or some other form of pre-democratic government). In the world, many societies found that democratic forms of government where just better than what they had, and so they changed.

Democracy was invented about 2500 years ago. It gave the ancient Greeks such a profound sociological advantage over other countries that they outcompeted all their local neighbors until, in short order, other countries were either adopting the system or being subsumed by them. Except, no, that didn't happen, they were dominated by various autocracies, and democracy vanished from the region for more than a millennium. The Roman Empire expanded far beyond the reaches of the Roman Republic.

Democracy has become so successful in the last few centuries not because provides countries with an innate competitive advantage, but because a) in recent history, some of the most powerful countries in the world have made a deliberate effort to export or impose democracy, and b) it's an appealing memeplex.

You can call this system "The absolute removal of a set of problems that has plagued humanity since there was a such thing as humanity," but that doesn't really say much about how useful it is. The eradication of sneezing would be the absolute removal of a problem that has plagued humanity since there was a such thing as humanity, but that doesn't mean it would be tremendously helpful. The specific examples you gave for how such a system would be useful were

Crime (Is someone going to jump out at me and steal my stuff or try to kill me?) Trustworthy Business Dealings (When I pay him, will he deliver the goods, or will he try to cheat me?) Public Works (We need this bridge built. If I throw in my share, how can I be sure everyone else will to?) Coups (If we assign this guy to be in charge of this important thing (maybe an army battalion), will he use it for the common good, or will he try to take over?) Treason (How can we know that our citizens are not secretly working for the enemy?)

In a country like, say, Finland, these aren't especially pressing problems. Yes, there's crime, but the rates are pretty low. It's not as if it subtracts even 10% of the country's productivity. When a Finnish person pays their taxes for public works, they don't have to worry that not enough people will pitch in and the work won't be paid for. A Finnish person can safely assume that when they deal with a business, they'll get the good or service they paid for. The chances of a coup in Finland are practically nil, and treason is not a significant danger.

These are problems that existing societies on earth have already managed to mostly solve. A better education system, where, for instance, every child of at least average intelligence comes out really understanding empiricism, rationality, and their own comparative advantage, is likely to be considerably more useful to such a society, without raising any tricky ethical issues.

But even without that, what proportion of wizardry jobs actually need magic to do? Most wizarding workers we have seen are shopkeepers, bureaucrats and the like, which don't really need magic in their activities. And yes, there has to be someone to make the magic items, but even if we assume that there are indeed out of sight magical sweatshops filled with workers making magic items, it would be silly to assume they make up more than 80% of the population, would it not?

Every character we've seen in the series uses magic in their daily lives. Even assuming that abilities such as being able to clean objects that would take several minutes of manual labor in a second with a spell, or organize a stockroom in seconds by waving a wand around, do not account for a large proportion of the productivity of the labor force, keep in mind that the magical abilities of the populace are largely responsible for the wizarding world being a "zeroth world country," as Harry puts it. The longevity of the populace, and much of their medical technology, relies on the innate healing and magical reservoirs of the magically gifted population. You're costing a significant proportion of the population about half their lifespans, in addition to a large portion of their quality of life (by losing their magical powers and having no right to mental autonomy, they're forced way, way down the totem pole status-wise, as well as being unable to access many of the conveniences of the magical population,) and a significant part of their productivity, in exchange for solving problems that other societies on earth have already managed to mostly solve.