that it's the same people being constantly harmed

Sure, but I don't see why you think the current setup is much better on that metric. Someone who consistently flubs interviews is going to be unemployed or underemployed, even though interviews don't seem to communicate much information about job productivity. If it were an actual lottery, I think the argument that the unemployment is spread evenly across the population would hold some weight, but I think employers have errors that are significantly correlated already, and I'm willing to accept an increase in that correlation in exchange for a decrease in the mean error.