Mostly Caplan, other papers (here's's two studies with commentary), and a prior determined mostly by a broader version of hereditarianism (seeing most of a person's 'quality' as fixed by reaching adulthood, so a combination of biological and early environmental effects). Just IQ is apparently 40%, add in Conscientiousness and it's likely to cross 50%, and now we're in the territory where it's mostly ability bias.

Perhaps this is implicit in what you say, but signaling and social networking benefits from going to an Ivy League may play major roles independently of human capital.