If you use the propensity score matching method, you begin by estimating the propensity score, then you match on the propensity score to create exposed and unexposed groups within levels of the propensity score. After you create those groups, there is a step where you can look at the matched groups without the outcome data, and assess whether you have achieved balance on the baseline covariates. If I understand Rubin's students correctly, they see this as a major advantage of the estimation method.
You can obviously blind yourself to the outcome using any estimation method, but I am not sure if there is a step in the process where you look at the data without the outcome to evaluate how confident you are in your work.
In order for the estimand of the propensity score method to be unbiased, the following is sufficient:
(a) SUTVA (this is untestable)
(b) Conditional ignorability (this is testable in principle, but only if we randomize the exposure A)
(c) The treatment assignment probability model (that is the model for p(A | C), where A is exposure and C is baseline covariates) must be correct.
It may be that the "balance property" tests a part of (b), but surely not all of it! That is, the arms might look balanced, but conditional ignorability might still not hold...
Students who go to more selective colleges make more money later in life. The 2013-2014 PayScale College Salary Report gives starting salaries of ~$60k for graduates of Ivy League schools vs. ~$45k/year for graduates of mid-tier state schools, and mid-career salaries of ~$110k/year for graduates of Ivy League schools vs. ~$80k/year for mid-tier state schools, so the difference is about 30%.
This partially reflects students who go to more selective colleges being more able and ambitious, as opposed to attending a more selective college boosting one's income. How much does attending boost income? The famous paper Estimating the Return to College Selectivity Over the Career Using Administrative Earning Data (2011) by Dale and Krueger raises the possibility that on average, attending a more selective college doesn't raise earnings at all. Specifically, controlling for
they found that as a group, there was no statistically significant difference in income later in life between students who went to more selective colleges and students who went to less selective colleges. Their finding is somewhat robust: it's based on a large (~10k) sample size, it's true both of the class of 1976 and the class of 1989, it's true of the class of 1976 from age 25 through age 50 and it's true both of men and of women.2,3
A couple of caveats:
Dale and Krueger's central finding has been taken to be evidence that going to a more selective college does not increase one's earnings, contrary to conventional wisdom.
There are many factors that could give rise to a difference in income between those who attended more selective colleges, some of which favor those who attended less selective colleges, and could counterbalance others that increase earnings. I list some potential contributing factors below (some of which were pointed out by Dale and Krueger). In each case, whether these effects are present is unclear, and to the extent that they are there, the overall effect of them is sometimes unclear, but they could give rise to a difference between the two groups.
Selection effects
These have no bearing on whether going to a more selective college increases expected earnings.
Cost
The cost of attending a more selective college could (after taking a student's ability and merit-based financial aid into account) be higher or lower than the cost of attending a selective college. So the choice of attending a more selective school could reflect having a family that's willing to pay more for college (and/or the willingness of the student for his or her family to pay).
Future career plans
Students who choose to attend more selective schools could be more or less likely to go into academia than students who don't. Academics make less money than other people do (after controlling for factors such as GPA, SAT scores and conscientiousness). Similarly, those who attend more selective schools may have other career preferences that feed into expected earnings
For students who go on to professional school (law, medicine or business), college attended doesn't matter as much for later life prospects. So attending a more selective college could reflect lower intent to go on to professional school. The earnings of people who go onto professional school are generally higher than those of people who don't – this points in the direction of expected earnings being higher for those who go to less selective colleges.
Concern for prestige
Students who choose to attend more selective colleges plausibly care more about prestige. This desire for prestige could correspond to more desire to make money later in life – this points in the direction of expected earnings being higher for those who go to more selective colleges.
Signaling
The impact of prestige of college attended on hiring decisions
According to research by Lauren Rivera, high paying elite professional service firms (investment banks, law firms, and management consulting firms) give a lot of weight to whether job applicants attended top four universities when making their hiring decisions.
More broadly, employers give weight to the prestige of college attended. However, the effect size is smaller than it might seem. In a 2013 Gallup Poll, 9% of business leaders said that the college a job applicant attended is "very important" to managers making hiring decisions, and 37% said that it's "somewhat important." Notably,
(i) employers listed college attended as the least important of the 4 factors that they were asked about
(ii) the American public gives more credence to employers weighting college attended when making hiring decisions: 30% think it's "very important" and 50% think it's "somewhat important."
(The 2013 data is much more recent than the Dale-Krueger data, but still provides relevant evidence.)
Once one is hired, where one went to college won't play a role in professional advancement at that firm (except to the extent that it built relevant skills).
Influence on grades
Carl Shulman suggested that going to a more selective school reduces one's expected GPA, because of higher grading standards. It's unclear to me whether this is true, and the effect could even cut the other way, but there's plausibly an effect in some direction. Reduced GPA reduces one's prospects for getting into medical or law school. Perusing forums for applicants, one finds many people saying that GPA is a dominant factor in admissions, one that's far more important than prestige of college attended.
Influence on major choice
Attending a more selective school reduces one's relative standing amongst students in a given major. This can lead to students at more selective schools choosing a less demanding major than they otherwise would have (whether because the major requires more effort than it would have, because one finds it intolerable to be one of the weaker students majoring in a given subject, or for some other reason). In the 2013 Gallup Poll cited above, business leaders surveyed listed the subject that a student majored in as significantly more important than college attended, in the context of hiring decisions.
Treatment effects
Peer group
Having a more capable peer group can lead to better learning opportunities, and higher expected earnings. Ben Kuhn wrote
It can also give one access to better advice: Ben Kuhn also wrote:
These considerations generally favor more selective schools, but not as strongly as might meet the eye: less selective schools often have honors courses and honors programs, where one might be able to meet students as capable as those who one would be interacting with at less selective colleges (though the best students at more selective colleges will generally be stronger than the best students at less selective colleges).
Networking benefits
Going to a more selective college will generally expose one to people who will be in higher places later on in life, and who will correspondingly be able to connect one with influential people in one's professional field, who may get one a high paying job and so forth. Such people may also serve as professional collaborators, for example, if one wants to do a startup right out of college.
As above, the effect here is smaller than might initially meet the eye, because one might be able to get similar benefits by interacting with the most capable students at a less selective college.
Confidence
Some people have suggested that being a "small fish in a big pond" reduces students' confidence relative to being a "big fish in a small pond." Assuming this, to the extent that confidence increase later life earnings, all else being equal, attending a more selective school will reduce expected earnings.
Better learning due to attention from professors
Some people have found that being a "big fish in a small pond" is conducive to getting more attention from the professors in one's classes, on account of standing out. This can increase the amount that a student learns, because of professors' greater willingness to spend time on personalized instruction.
Self-sufficiency
Some people who I know report to having a subjective sense that being in a less elite environment was helpful to them, because it forced them to be independent (on account of being different from their peers), whereas had they been in an elite environment, they would have "gone with the flow" and uncritically made their decisions based on what their peers were doing. All else being equal, this factor would increase expected earnings of those who attend less selective colleges.
Influence on major choice
Influence on career choice
Going to a more selective college could nudge one toward or away from academia, going into the non-profit world, getting a professional degree, etc. Earnings vary across these fields, and this could give rise to a difference between the groups.
Implications
Before thinking seriously about the paper by Dale-Krueger, I had subscribed to the conventional wisdom that going to a more selective college generally boosts expected earnings. Thinking it over more carefully, it's now genuinely unclear to me whether this is the case, and it could reduce expected earnings in general.
Rather than taking the Dale-Krueger finding to be definitive, one should give some weight to conventional wisdom, on the grounds that the paper might have hidden methodological errors, or have ceased to be relevant in the present day. But in view of
it seems reasonable to adopt a "best guess" that if going to a more selective college does increase expected earnings, the effect size isn't high.
What implications does this have for students who are trying to decide what college to go to, or how much to focus on getting into college? First some general considerations:
Footnotes
[1] You might wonder why the authors didn't control for the average SAT score of the colleges at which the students were accepted. The authors did something like this (actually, comparing students who had been accepted at the exact same set of colleges) in a 2002 paper, and obtained similar results. From the paper: "The matched applicant model and self-revelation model yielded coefficients that were similar in size, but the self-revelation model yielded smaller standard errors. Because of the smaller sample size in the present analysis, we therefore focus on the self-revelation model."
[2] The authors say this in the text of the paper, but when I look at Table 4 of the paper on page 32 of the PDF, the data seems to indicate that for women from the 1976 cohort broken down into age groups, there is a statistically significant difference, in favor of those who attended less selective schools. But I assume that I'm misinterpreting the table: I'd welcome any help interpreting the data on this point.
[3] The authors qualify this by saying "The estimates from the selection-adjusted models are imprecise, especially for the 1989 cohort. Thus, even though the point-estimates for the return to school quality are close to zero, the upper-bound of the 95 percent confidence intervals for these estimates are sometimes sizeable."
Cross-posted from the Cognito Mentoring blog