Harvard has two responses to these claims: they can continue getting called racist, or they can unfairly admit minorities who don't deserve to get in. Both of these alternatives are preferable to telling a large group with normal human psychology that they are inferior.

Edit: I tend to think it's OK for Harvard to admit too many minorities because I think having the upper class be made up of people from a variety of backgrounds is valuable. But I could see how this could be a problem in other domains, such as cousin_it's mortgage example.