I asked my father to read this and give his thoughts.
He says that positive selection only works well when you have a very good idea what you need to select for. If you're sending an athlete to the Olympics but the event he'll have to compete in will be chosen at random, you can't just choose the one with the best time on the 800 meter dash, because the event might end up being something like archery, fencing, or weightlifting. And you certainly wouldn't want to send a non-swimmer. If you need a generalist, seeing how well someone does at jumping through a wide variety of arbitrary hoops might really be the best test you can practically implement.
(Now I'm wondering just how good or bad the 800 meter dash actually is at predicting levels of success at unrelated sports. For example, could you tell the difference between an NHL-quality ice hockey player and one that plays on a minor league team just by looking at their times on the 800 meter dash?)
Assuming a significantly large distribution of athletes sent by other rational managers, where all athletes are bound to the same rules of random event selection, I would still send the best possible specialist in a single discipline in this case, because without certainty that all other rational managers know certainly that some generalists will be better in everything than other generalists and that each one is confident that theirs is best, I conclude that some of them attempt a gamble of probabilities and send a specialist, and thus I also send a specialist to maximize my chances of winning.
After all, there are higher chances of the event being my athlete's specialty than there are chances of every single other athlete being less good at it if I pick a generalist, unless the number of possible events is large enough to outweigh the number of athletes. Throw in irrational managers and the possibility of other managers having information unavailable to you, and your father's argument seems very weak.
Now, of course, I'm probably attacking something that wasn't meant to be a strong defensible argument. However, I feel very strongly about the point that negative selection is wrong i...
Well, the sports analogy was my own interpretation of what he said.
Game theory question time: you and N other players are playing a dice rolling game. Each player has the choice of rolling a single twenty-sided die, or rolling five four-sided dice. The player with the highest total wins. (Ties are broken by eliminating all non-tying players and then playing again.) Now, rolling 5d4 has an expected score of 12.5 and rolling 1d20 has an expected score of 10.5, so when N=2, it's obviously better to roll 5d4. However, when N becomes sufficiently large, someone is going to roll a 20, so it's better to pick the 20-sided die, which gives you a 1 in 20 chance of rolling a 20 instead of a 1 in 1024 chance of getting five 4s. For exactly what value of N does it become better?
Edit: Fixed stupid math mistakes. That'll teach me to post after staying up all night!
One of the most important social structures of modern society is the corporation - a framework for large groups of people to band together and get absolutely huge projects done. In this framework, the structure itself is more important than individual excellence at most levels. To a lesser extent, the same applies to academia and even "society as a whole".
In that context, I think preferring negative selection to positive makes sense: a genius data-entry clerk is less helpful than an insubordinate data-entry clerk is disruptive.
And remember that we have side routes so real geniuses (of some kinds) can still make it: set up their own company, start their own political party, start publishing their work online, design games in their basement, and so on.
And remember that we have side routes so real geniuses (of some kinds) can still make it: set up their own company, start their own political party, start publishing their work online, design games in their basement, and so on.
This is a really good point. It's good to have low barriers to this sort of thing. For instance, if you need to hire a lawyer and an accountant to set up your own company, then a genius cookie baker can't set up their own cookie shop unless they also have the money or connections to get the help of a lawyer and an accountant.
Really good post...it makes a point that is completely new to me, which is always nice.
It does occur to me that the current (negative selection) system would reward "hard work" more, relative to "talent", than a positive selection system. (In quotation marks because those are both metrics that are hard to measure separately from one another.) Someone who is very conscientious and hard-working is likely to compensate for wherever areas they're weaker, in terms of "natural talent", however you define that.
My first, emotional reaction to your post was "I would be screwed in a positive selection system!" As someone who's above average in a lot of areas, not really exceptional in any, and obsessively hard-working enough that it's a running joke among my friends, I like the current system just fine (although I'm not in academia.) I don't know if conscientiousness would have a bigger long-term effect on results than innate brilliance; it probably depends on what field you're talking about.
My intuition says that a positive selection system would probably be a good idea in fields where there is big variance in natural ability, i.e. math or physics, and less so in fields like medicine where a lot of "talent" depends on how willing you are to work hard and keep improving over your whole career.
Perhaps this should influence my career decision somewhat, its hard to tell if talent or effort is more crucial for programming.
Effort. Always assume effort. Talent will speed up the learning process in the early stages, is likely to make effort easier (because it is more fun) and at the extreme upper ends of of performance probably gives a higher limit. But in general effort plus social politics skill will determine your career success.
Despite what they are taught likely to be about themselves, what they might think of themselves, and what western culture expects of them, programmers are more creative artists than analytic engineers.
The difference is most tangible from the management perspective since motivating programmers is less like motivating chemical, mechanical, or any other sort of engineer and more like motivating commercial artists with less pretense, who were never led to believe they were meant for something greater. Dissatisfaction from programmers grows in much the same way it grows in commercial artists as well, though they programmer is less likely to specifically identify his or her complaint and the artist is more likely to complain about having sold his or her soul.
Common responses to criticism of work among programmers align more with those among artists than those among engineers. Again, I learned this from a managerial perspective.
The most important advice that may be given to starting artists (excluding all the low-hanging fruit advice that is best for everyone in general, of course) isn't about discovering your own inner talent or anything similar, instead it is about discipline: "Ideas are not swords you can brandish about in triumph. What matters most is the Sit Down, Shut Up And Get It Done. Only there will you find the true steel for your craft. Only there, will you know if you are worth the words out of your mouth."
The concepts of positive and negative selection are not quite well defined in your essay, I think.
Imagine that you have one test, with a gaussian distribution of outcomes. Let's arbitrarily set a threshold, and if people are above this threshold, they have passed this test. Call the sets of passing A and not passing ~A
Would you call this a positive or negative selection? It is neither, in my opinion.
Now, imagine you have two tests, A and B.
A positive test is one where A U B are selected. A negative test is one where ~(~A ^ ~B) are selected.
In other words - the operative difference between positive and negative selection is OR vs. AND.
I've just realized that I have been treating dating as a negative selection process. This might explain the lack of success.
If A means not B, then selecting for A is the same thing as selecting against B.
If A means "with probability 90% not B", then if B is a serious problem, it is worth checking both A and not B. Maybe even checking not B first, to avoid halo effect from A.
In my experience, some people treat dating as a negative selection process with thousand requirements that no one passes, because thousand criteria are simply too much. (Assuming independent results, even with probability 99% of passing each test, less than one person in 20 000 passes all thousand criteria. In real life, the criteria are often positively correlated, but on the other hand the probability is way less than 99%.) And those people usually defend it by taking each criterium out of the context and saying: "What's wrong about wanting my boyfriend/girlfriend to be interested in opera/programming?" Well, nothing wrong per se, but if you have thousand criteria like this, good luck finding a person who fulfills them all (and is also interested in you).
The solution is to separate those criteria into two groups: "must have" and "nice to have". (And if nine hundred of the thousand criteria ar...
Grades in high school are already like this. To get the best grade from a subject, you need to be good, but not exceptional. So being good but not exceptional in everything brings you the highest possible score.
If instead you are exceptional in a subject or two, and average in a few unrelated subjects, it gives you lower total score, and if the university cared about your grade average, you would have problems getting there, especially if many people with the highest total score competed with you.
I don't think it's quite true that "fail once, fail forever", but the general point is valid that our selection process is too much about weeding-out rather than choosing the best. Also, academic doesn't seem to be very good at the negative selection that would make sense, e.g. excluding people who are likely to commit fraud or who have fundamentally anti-scientific values. (Otherwise, how can you explain how Duane Gish made it through Berkeley?)
Since most production functions are quasiconcave over inputs, negative selection is a cheap method of increasing expected return. You lose some outliers and also people who would be good in those rare domains with quasiconvex production functions, but our system is optimized for the average case.
In the college admissions example, a top school wants to admit undergraduates likely to become successful doctors/lawyers/businesspeople and alumni donors, not gamble that the smart kid with a few Bs in high school is going to revolutionize a scientific field in 1...
I agree with most of it, though the point about academia is a bit contrived.
True, there is a lot of negative selection before you get a cushy job the usual way, but you can certainly bypass quite a few obstacles if you are exceptionally good. For example, solve any of the open problems in math or physics, post a preprint on arxiv.org (well, you may need someone to vouch for you, but that's not really an issue) and you are all set.
Unfortunately, I cannot recall a single discovery in physics in the last half a century that was not made by someone who jumped...
While [negative selection] filters out some good people, it probably does not reject the very best, otherwise we would see an occasional example of someone making a significant discovery outside academia.
I predict that we will indeed see this before too long, now that we have the internet; and it will thus turn out that some of the best people were being filtered out. Access to information and social support/reinforcement is a huge limiting factor.
And of course, if you're willing to look a century back instead of just a half-century, you find the salient example of Einstein -- who didn't even have the internet, but still managed to advance science from outside the "establishment" (which was a sizable apparatus in his time and place, just as it is in ours).
What my externally observable percentiles look like:
What my educational credentials look like:
Programming: 95%
I'm with J and Alex -- are you comparing yourself to people or to programmers? I'm fairly sure FizzBuzz puts one well above 95% in the general population
That'd be a much harder question to answer; my talent is specialized toward figuring out the shape of the right theorem to be proved, not the actual proof which is where most modern math concentrates its prestige. (This is an objectively verifiable form of mathematical talent; it means that sometimes Marcello would prove something and I would look at it and say, "That doesn't look right" and at least half the time there'd be a mistake.) I feel insecure about not being an expert in the tools by which most modern mathematicians measure basic competence; I can also apparently make "well, if that's your problem, try transforming it this way" suggestions to someone doing graduate mathematical research at Yale that are accepted as brilliant. I confidently depose that, even taking unusual talents into account, I am not in the literal top tier of mathematical potential - if I can explain basic Bayes better than anyone or was first to state the Lob problem or invented TDT, those outputs drew on at least some non-mathematical high-percentile sections of my brain (explanatory ability in the first case, or what's somewhat vaguely referred to as "philosophical" ...
That'd be a much harder question to answer
"What did you mean when you said 99%" should not be a hard question to answer. "Which alternative is correct" may be hard, but did you have in mind "one alternative is correct but I don't know which"?
Are these your own estimates, or have you found some objective, accurate test for ranking "Conceptual originality"?
Students at Yale are, for the most part, all strikingly similar – same socioeconomic class, same interests, same pursuits, same life goals, even the same style of dress.
Can I ask on what basis you're drawing this conclusion? I agree with the bulk of what you said about overuse of negative selection, but I challenge the idea that it's producing cookie-cutter student bodies at elite universities. Having attended Yale as an undergrad, your claim strikes me as incorrect, as the Yale student body seemed to me more diverse on all five of those categories than...
I've never understood the reason for giving grades A-E or fail, like we do for O and A levels, or I:II:III:fail, like we do for degrees.
My father's O-levels gave a percentile ranking, so he was e.g. in the 83rd percentile in the country for history.
So we must have changed over at some point. Does anyone know why? It's always looked like throwing information away to me, and it's also unfair to people on the grade boundaries.
Of course this may be motivated thinking on my part, I'd much rather have had a string of 100s for my exams than a string of As, and I'd much prefer to have got a 75 for my degree than a II (which covered percentiles 25-75) !!
There’s no one way you can do really, really well, and thereby be admitted to Harvard.
You don't think Jack Andraka will get admitted to Harvard because he did one thing really, really well?
I got into UC Berkeley with a high school GPA of 2.9 by talking about math with professors. This strategy failed everywhere else, and would have failed at Berkeley if I hadn't been lucky enough to find a professor stubborn enough to argue with the admissions office again after they ignored him the first time. On the other hand, my accomplishments are not even close to as impressive as Andraka's, so he might have an easier time with this strategy even with a worse GPA.
Anyway, if you've done anything impressive, finding a champion within the system is easy. Andraka had a hard time with that step because he was trying to get support before doing something cool rather than after. Now, the vast majority of biology professors would gladly stand up for him to their institution's admissions department. But this strategy requires persistence on the part of the champion, as well as the applicant.
I have a dean of admissions at a large university in my nuclear family. Eliezer is right, there's no list like this.
But on the other hand, "Intel Science Fair winner" will PROBABLY attract the attention of the admissions committee. It's basically up to the applicant to craft a good applications package (including essay and letters of recommendation) that will capitalize on their amazing, singular strength, and throw weaknesses like GPA into shadow. If the applicant can't do this, they won't be admitted.
It's a Bardic Conspiracy problem, really. It's a matter of storytelling and presentation.
I note that it also makes no sense to filter excellent scientists who aren't good writers or who take a long time to write (e.g. PhD dissertation test). If you can do the research, someone else should be able to specialize in writing for you.
It's remarkable how many barrier-to-entry professions revolve around the denial of professional specialization. A surgeon can't just be someone of moderate intelligence and exceptional dexterity who studies and practices one key surgery, no, they need 11 years of medical school that they'll mostly never use. A scientist is forced to write. And so on.
I just now read an interview which brings up the rise of negative selection in job applications:
...In the past, they wanted lots of applicants, so now they’re overwhelmed by applicants, so now every company will tell you they’re getting thousands or tens of thousands of applicants for positions. You couldn’t possibly screen them all by hand, because you can’t look at them all, so they use automated systems to do the screening. But the screening is never as good as somebody who has human judgment, and the way screening works is you build in a series of typic
It’s apparently so important that people really care about performance – as opposed to, say, in medicine, where we exclude brilliant doctors if they don’t have the stamina to work ninety hours a week.
How much does this actually matter, I wonder? Is there really that big a difference between the best doctor in a group of 100 and the 10th best doctor in that same group? (The 10th best golfer in a tournament doesn't take home the trophy, but the 10th best doctor in the hospital can still do a fine job treating a broken arm.)
In some examples, like elite college admission, it seems more like there are both negative and positive controls. Although colleges use SAT scores and GPAs to weed out people who aren't "good enough," they also look at whether students are exceptional through supplemental essays or awards the student received in high school. Negative controls bar many students from admission, but in many cases, positive controls must also be used to select a final class out of acceptable applicants. I'm not in academia, but it seems similar. Although you may get ...
Academia is another example of negative selection.
This doesn't seem like a fair generalization. At the undergraduate level, selection procedures are (and are rightly) negative. At the graduate level, things are very different across different fields and departments, but on the whole I think graduate admissions is a mix: people are weeded out until you have a small group of acceptable candidates, and then the exceptional ones are pulled out on the basis of their specific work. At the tenure level, you get a similar mix but there it's very heavily in the ...
Very thought-provoking. Thank you!
How would we tell the difference between 'positive' and 'negative' selection? If I imagine that I've got two scores x and y (00.9 AND y>0.9' what you mean by 'negative selection', and 'accept if x>0.995 OR y>0.995' positive selection?
If I'm thinking along the right lines here, is there a general principle (like 'acceptance set must be convex'?), or do I have the wrong end of some crucial stick?
One word:
China.
One massive examination that determines your entire future? Isn't that about as positive selection as you can get?
Elite college admissions is an example of a negative selection test. There’s no one way you can do really, really well, and thereby be admitted to Harvard.
I suspect that "being rich enough to make a sufficiently large donation" will get you in (as long as you've got a high school diploma or GED). "Sufficiently large" may run in the hundreds of millions of dollars, though.
(Originally posted to my blog, The Rationalist Conspiracy; cross-posted here on request of Lukeprog.)
You’re the captain of a team, and you want to select really good players. How do you do it?
One way is through what I call positive selection. You devise a test – say, who can run the fastest – and pick the people who do best. If you want to be really strict, like if you’re selecting for the Olympics, you only pick the top fraction of a percent. If you’re a player, and you want to get selected, you have to train to do better on the test.
The opposite method is negative selection. Instead of one test to pick out winners, you design many tests to pick out losers. You test, say, who can’t run very well when it’s hot out, and get rid of them. Then you test who can’t run very well when it’s cold out, and get rid of them. Then you test running in the rain, and get rid of the losers there. And so on and so forth. When you’re strict with negative selection, you have lots and lots of tests, so that it’s very hard for any one person to pass through all the filters.
I think a big part of where American society’s gone wrong over the last hundred years is the ubiquitous use of negative selection over positive selection. (Athletics is one of the only exceptions. It’s apparently so important that people really care about performance – as opposed to, say, in medicine, where we exclude brilliant doctors if they don’t have the stamina to work ninety hours a week.) A single test can always be flawed; for example, IQ tests and SATs have many flaws. However, with negative selection, how badly you do is determined by the failure rate of every test combined. If you have twenty tests, and even one of them is so flawed it excludes good players, then your team will suck.
Elite college admissions is an example of a negative selection test. There’s no one way you can do really, really well, and thereby be admitted to Harvard. Instead, you have to pass a bunch of different selection filters: Are your SATs good enough? Are your grades good enough? Is your essay good enough? Are your extracurriculars good enough? Are your recommendations good enough? Failure on any one step usually means not getting admitted. And as competition has intensified, colleges have added more and more filters, like the supplemental applications top schools now require (in addition to the Common Application). It wasn’t always this way – Harvard used to admit primarily based on an entrance exam – until they discovered this let too many Jews in (no, seriously). More recently, the negative selection has been intensified by eliminating the SAT’s high ceiling.
Academia is another example of negative selection. To get tenure, first you have to get into a top PhD program. Then you have to graduate. Then you have to get a good recommendation from your advisor. Then you have to get a good postdoc. Then you have to get another good postdoc. Then you have to get a good assistant professorship. Then you have to get approved by the tenure committee. For the most part, if even one of those steps goes wrong – if you went to a second-tier PhD program, say – there’s no way to recover. Once you’re off the “track”, you’re off, and there’s no getting back on. It’s fail once, fail forever.
Grades are another example – A is a good grade, but there’s no excellent grade. There’s no grade that you only get if you’re in the top 0.1%. Hence, getting a really good GPA doesn’t mean excelling, so much as it means never failing. If you’re in high school and are taking six classes, if you fail one, your GPA is now 3.3 or less, regardless of how good you are otherwise.
In any field, at the top end, you tend to get a lot of variance. (Insert tales of the mad artist and mad mathematician.) Negative selection suppresses variance, by eliminating many of the dimensions on which people vary. Students at Yale are, for the most part, all strikingly similar – same socioeconomic class, same interests, same pursuits, same life goals, even the same style of dress. A lot of people tend to assume performance follows a bell curve, but in some cases, it’s more like a Pareto distribution: the top people do hundreds or thousands of times better than average. Hence, if you eliminate the small fraction of people at the very top, your performance is hosed. Fortunately for VC funds, the startup world is still positive selection.
Less obviously, a world with lots of negative selection might be a nasty one to live in. If you think of yourself as trying to eliminate bad, rather than encourage good, you start operating on the purity vs. contamination moral axis. Any tiny amount of bad, anywhere, must be gotten rid of, and that can lead to all sorts of nastiness. “When you are a Guardian of the Truth, all you can do is try to stave off the inevitable slide into entropy by zapping anything that departs from the Truth. If there’s some way to pump against entropy, generate new true beliefs along with a little waste heat, that same pump can keep the truth alive without secret police.”