Competition is generally not a competition in potential skill, but a competition in actual skill.
Here we get back to the point of my first comment.
Suppose that you have two groups, the Blues and the Greens, which have a huge gulf in economic status, but equal intrinsic abilities. Blues are almost all put through excellent, expensive schooling, whereas Greens are haphazardly educated. When they compete for access to higher education, Blues tend to overwhelmingly outperform Greens. Future Greens are left in a disadvantaged position similar to their parents.
In a sense, the Blues and Greens are competing on equal footing; if they receive the same results, they get the same access. But the affirmative action position is that, given the opportunity to correct the situation, it's unfair to leave the Greens in a cycle where they'll continually receive less opportunity relative to their aptitude.
I don't contest that affirmative action as currently applied fails to be an honest accounting of our best estimates of potential-given-disadvantage. I wouldn't even assert that our current affirmative action system is the best we can do with our current budget without crossing existing political taboos. But some of the factors you're suggesting accounting for are difficult and expensive to track and/or subject to too much political stigma to feasibly implement.
If you want to track variable X, and you have two heuristics, A and B, where A has 85% accuracy and B has 99% accuracy, but is a huge political liability to anyone who employs it, then for practical purposes your choice is between using heuristic A, or not tracking variable X.
The level of abstraction obfuscates the issue. The problem is that some people want a racial spoils system.
People generally approve of means tested public assistance, particularly for children, while many disapprove of race based public assistance. White is not a synonym for advantaged, and black is not a synonym for disadvantaged. Help the disadvantaged. Easy peasy.
If a greater percentage of blacks are disadvantaged, a greater percent can qualify for assistance, this time without the stigma associated with race based preferences.
Of course, the cost a pr...
Related: Heuristics for Evaluating the Soundness of the Academic Mainstream, Admitting to Bias, The Ideological Turing Test