gwern comments on LW Women: LW Online - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (590)
Sure, there could be non-monotonicity.
Imagine that Harvard lets in equal numbers of 'athletes' and 'nerds', the 2 groups are different populations with different means, and they do something like pick the top 10% in each group by score. Clearly there's going to be a bimodal histogram of SAT scores: you have a lump of athlete scores in the 1000s, say, and a lump of nerd scores in the 1500s. Sure. 2 equal populations, different means, of course you're going to see a bimodal.
Now imagine Harvard gets more 10x more nerd applicants than athletic applicants; since each group gets the same number of spots, a random nerd will have 1/10 the admission chance as an athlete. Poor nerds. But Harvard kept the admission procedure the same as before. So what happens when you look at admission probability if all you know is the SAT score? Well, if you look at the 1500s applicants, you'll notice that an awful lot of them aren't admitted; and if you look at the 1000s applicants, you'll notice that an awful lot of them getting in. Does Harvard hate SAT scores? No, of course not: we specified they were picking mostly the high scorers, and indeed, if we classify each applicant into nerd or athlete categories and then looked at admission rates by score, we'd see that yes, increasing SAT scores is always good: the nerd with a 1200 better apply to other colleges, and the athlete with 1400 might as well start learning how to yacht.
So even though in aggregate in our little model, high SAT scores look like a bad thing, for each group higher SAT scores are better.
Reminds me of Simpson's paradox.
Yes, I don't think we could make a conclusive argument against the claim that SAT scores may not help at all levels, not without digging deep into all the papers running logistic regressions; but I regard that claim as pretty darn unlikely in the first place.
They could be self-delusive, doing it to appease a delusive parent ('My Johnnie Yu must go to Harvard and become a doctor!'), gambling that a tiny chance of admission is worth the effort, doing it on a dare, expecting that legacies or other things are more helpful than they actually are...
Sure, maybe you can make a model that outputs Harvard or Princeton's results, but how do you explain the difference between Harvard and Princeton? It is easier to get into Princeton as either a jock or a nerd, but at 98th SAT percentile, it is harder to get into Princeton than Harvard. These are the smart jocks or dumb nerds. Maybe Harvard has first dibs on the smart jocks so that the student body is more bimodal at other schools. But why would admissions be more bimodal? Does Princeton not bother to admit the smart jocks? That's the hypothesis in the paper: an SAT penalty. Or maybe Princeton rejects the dumb nerds. It would be one thing if Princeton, as a small school, admitted fewer nerds and just had higher standards for nerds. But they don't at the high end. What's going on? Here's a hypothesis: Harvard (like Caltech) could admit nerds based on other achievements that only correlate with SATs, while Princeton has high pure-SAT standards.
I don't think an SAT penalty is very plausible, but nothing I've heard sounds plausible. Mostly people make vague models like yours that I don't think explain all the observations. The hypothesis that Princeton in contrast to Harvard does not count SAT for jocks beyond a graduation threshold at least does not sound insane.
I take graphs over regressions, any day.
Regressions fit a model. They yield very little information. Sometimes it's exactly the information you want, as in the calculation you originally brought in the regression for. But with so little information there is no possibility of exploration or model checking.
By the way, the paper you cite is published at a journal with a data access provision.
Dunno. I've already pointed out the quasi-Simpsons Paradox effect that could produce a lot of different shapes even while SAT score increases always help. Maybe Princeton favors musicians or something. If the only reason to look into the question is your incredulity and interest in the unlikely possibility that increase in SAT score actually hurts some applicants, I don't care nearly enough to do more than speculate.
I have citations in my DNB FAQ on how such provisions are honored mostly in the breach... I wonder what the odds that you could get the data and that it would be complete and useful.