affirmative action already takes different base expectations into account (if you live in a city of 50% black people and 50% white people, but 10% of local lawyers are black, your local law firm isn't required to have 50% black lawyers, but 10%).

I didn't know this. Could someone please give me a link confirming that affirmative action quotas depend on specific city?

Still, I can imagine a city where most members from some minority live on one side of the city, and your company is on the other side of the city, and you will be called racist simply because people don't like to commute to the opposite side of the city.