Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Negative and Positive Selection

71 Post author: alyssavance 06 July 2012 01:34AM

(Originally posted to my blog, The Rationalist Conspiracy; cross-posted here on request of Lukeprog.)

You’re the captain of a team, and you want to select really good players. How do you do it?

One way is through what I call positive selection. You devise a test – say, who can run the fastest – and pick the people who do best. If you want to be really strict, like if you’re selecting for the Olympics, you only pick the top fraction of a percent. If you’re a player, and you want to get selected, you have to train to do better on the test.

The opposite method is negative selection. Instead of one test to pick out winners, you design many tests to pick out losers. You test, say, who can’t run very well when it’s hot out, and get rid of them. Then you test who can’t run very well when it’s cold out, and get rid of them. Then you test running in the rain, and get rid of the losers there. And so on and so forth. When you’re strict with negative selection, you have lots and lots of tests, so that it’s very hard for any one person to pass through all the filters.

I think a big part of where American society’s gone wrong over the last hundred years is the ubiquitous use of negative selection over positive selection. (Athletics is one of the only exceptions. It’s apparently so important that people really care about performance – as opposed to, say, in medicine, where we exclude brilliant doctors if they don’t have the stamina to work ninety hours a week.) A single test can always be flawed; for example, IQ tests and SATs have many flaws. However, with negative selection, how badly you do is determined by the failure rate of every test combined. If you have twenty tests, and even one of them is so flawed it excludes good players, then your team will suck.

Elite college admissions is an example of a negative selection test. There’s no one way you can do really, really well, and thereby be admitted to Harvard. Instead, you have to pass a bunch of different selection filters: Are your SATs good enough? Are your grades good enough? Is your essay good enough? Are your extracurriculars good enough? Are your recommendations good enough? Failure on any one step usually means not getting admitted. And as competition has intensified, colleges have added more and more filters, like the supplemental applications top schools now require (in addition to the Common Application). It wasn’t always this way – Harvard used to admit primarily based on an entrance exam – until they discovered this let too many Jews in (no, seriously). More recently, the negative selection has been intensified by eliminating the SAT’s high ceiling.

Academia is another example of negative selection. To get tenure, first you have to get into a top PhD program. Then you have to graduate. Then you have to get a good recommendation from your advisor. Then you have to get a good postdoc. Then you have to get another good postdoc. Then you have to get a good assistant professorship. Then you have to get approved by the tenure committee. For the most part, if even one of those steps goes wrong – if you went to a second-tier PhD program, say – there’s no way to recover. Once you’re off the “track”, you’re off, and there’s no getting back on. It’s fail once, fail forever.

Grades are another example – A is a good grade, but there’s no excellent grade. There’s no grade that you only get if you’re in the top 0.1%. Hence, getting a really good GPA doesn’t mean excelling, so much as it means never failing. If you’re in high school and are taking six classes, if you fail one, your GPA is now 3.3 or less, regardless of how good you are otherwise.

In any field, at the top end, you tend to get a lot of variance. (Insert tales of the mad artist and mad mathematician.) Negative selection suppresses variance, by eliminating many of the dimensions on which people vary. Students at Yale are, for the most part, all strikingly similar – same socioeconomic class, same interests, same pursuits, same life goals, even the same style of dress. A lot of people tend to assume performance follows a bell curve, but in some cases, it’s more like a Pareto distribution: the top people do hundreds or thousands of times better than average. Hence, if you eliminate the small fraction of people at the very top, your performance is hosed. Fortunately for VC funds, the startup world is still positive selection.

Less obviously, a world with lots of negative selection might be a nasty one to live in. If you think of yourself as trying to eliminate bad, rather than encourage good, you start operating on the purity vs. contamination moral axis. Any tiny amount of bad, anywhere, must be gotten rid of, and that can lead to all sorts of nastiness. “When you are a Guardian of the Truth, all you can do is try to stave off the inevitable slide into entropy by zapping anything that departs from the Truth.  If there’s some way to pump against entropy, generate new true beliefs along with a little waste heat, that same pump can keep the truth alive without secret police.”

Comments (262)

Comment author: Ezekiel 05 July 2012 06:42:04PM 17 points [-]

One of the most important social structures of modern society is the corporation - a framework for large groups of people to band together and get absolutely huge projects done. In this framework, the structure itself is more important than individual excellence at most levels. To a lesser extent, the same applies to academia and even "society as a whole".

In that context, I think preferring negative selection to positive makes sense: a genius data-entry clerk is less helpful than an insubordinate data-entry clerk is disruptive.

And remember that we have side routes so real geniuses (of some kinds) can still make it: set up their own company, start their own political party, start publishing their work online, design games in their basement, and so on.

Comment author: fubarobfusco 05 July 2012 11:43:21PM 10 points [-]

And remember that we have side routes so real geniuses (of some kinds) can still make it: set up their own company, start their own political party, start publishing their work online, design games in their basement, and so on.

This is a really good point. It's good to have low barriers to this sort of thing. For instance, if you need to hire a lawyer and an accountant to set up your own company, then a genius cookie baker can't set up their own cookie shop unless they also have the money or connections to get the help of a lawyer and an accountant.

Comment author: DjangoCorte 10 July 2012 07:17:56PM 1 point [-]

I'm not sure that it's the corporate structure that makes negative selection more useful in the data entry case. It's not the fact that the data-entry clerk is part of a large organisation that means that a slightly incompetent data-entry clerk is more disruptive than a genius-level one is helpful. Rather it's the fact that data-entry is a relatively low skill job and with relatively little room for excelling above mere competence. Leaving the corporation wholly out of it, and imagining a person doing data entry in complete isolation, the most helpful data-entry clerk would still be selected by making sure they weren't terrible, but weren't necessarily brilliant, at typing and remaining attentive etc. I think this idea is supported by the fact that for higher level/skill positions, one probably would want to employ more positive selection.

If your point was specifically that insubordination (and not just slight incompetence in general) is more harmful than genius-level work is helpful, then I guess that, in an obvious sense, the harm of insubordination is due to the corporate nature of work (since you can't be insubordinate outside of a group hierarchy). But then I'm not sure that insubordination-worries requires negative selection, or at least not a wide range of negative selection tests. Sure, you might want to include a negative selection test along the lines of 'are they likely to do the opposite of what they're told on a whim occasionally?', but it's an open question whether the rest of your criteria would be negative or positive.

Comment author: Ezekiel 10 July 2012 09:45:37PM 0 points [-]

The point I should have made clear was that data-entry clerks don't exist outside of corporations, because in isolation they're useless. More generally, mass production has been made possible by the production-line paradigm: break down the undertaking into tiny discrete jobs and assign a bunch of people to doing each one over and over again.

Once you get that kind of framework, exceptionally good workers aren't very helpful, because the people to either side of them in the production line aren't necessarily going to keep up. You just need to shut up and do your job, the same as everyone else.

At the high levels - the people wielding their collective underlings as a tool, rather than the people who are part of that tool - this obviously no longer works.

Important note: all of the above, including my original comment, is 100% psuedo-intellectual wank, since I've never been part of a corporation, never taken a business management course or seminar, and never conducted or read a study on the efficacy of various business practices.

Comment author: Yvain 05 July 2012 02:36:54AM 9 points [-]

This was my favorite post on your blog and I'm glad you posted it here.

Comment author: Daermonn 05 July 2012 06:10:16PM 2 points [-]

I agree. I stumbled across this one a week ago or so - without knowing the author was associated with LW - loved it, and have been thinking about it off and off since. I'm glad to see it again. I feel like I should probably start reading your blog regularly..

Comment author: Swimmer963 05 July 2012 04:57:12PM 8 points [-]

Really good post...it makes a point that is completely new to me, which is always nice.

It does occur to me that the current (negative selection) system would reward "hard work" more, relative to "talent", than a positive selection system. (In quotation marks because those are both metrics that are hard to measure separately from one another.) Someone who is very conscientious and hard-working is likely to compensate for wherever areas they're weaker, in terms of "natural talent", however you define that.

My first, emotional reaction to your post was "I would be screwed in a positive selection system!" As someone who's above average in a lot of areas, not really exceptional in any, and obsessively hard-working enough that it's a running joke among my friends, I like the current system just fine (although I'm not in academia.) I don't know if conscientiousness would have a bigger long-term effect on results than innate brilliance; it probably depends on what field you're talking about.

My intuition says that a positive selection system would probably be a good idea in fields where there is big variance in natural ability, i.e. math or physics, and less so in fields like medicine where a lot of "talent" depends on how willing you are to work hard and keep improving over your whole career.

Comment author: pnrjulius 06 July 2012 10:54:30PM 0 points [-]

Negative selection may be good, actually, for the vast majority of people who are ultimately going to be mediocre.

It seems like it may hurt the occasional genius... but then again, there are a lot more people who think they are geniuses than really are geniuses.

Comment author: [deleted] 05 July 2012 05:14:48PM 0 points [-]

My first reaction was pretty much identicle, right now you can do well at almost anything purely based on conscientiousness, including video games, work, school, and social interaction. I don't know of any good way to measure general talent, but when I learn most things I tend to be quite bad at them until I enter tsukoku naritai mode. Perhaps this should influence my career decision somewhat, its hard to tell if talent or effort is more crucial for programming.

Comment author: wedrifid 05 July 2012 08:34:43PM 13 points [-]

Perhaps this should influence my career decision somewhat, its hard to tell if talent or effort is more crucial for programming.

Effort. Always assume effort. Talent will speed up the learning process in the early stages, is likely to make effort easier (because it is more fun) and at the extreme upper ends of of performance probably gives a higher limit. But in general effort plus social politics skill will determine your career success.

Comment author: Never_Seen_Belgrade 06 July 2012 04:52:24AM *  7 points [-]

Despite what they are taught likely to be about themselves, what they might think of themselves, and what western culture expects of them, programmers are more creative artists than analytic engineers.

The difference is most tangible from the management perspective since motivating programmers is less like motivating chemical, mechanical, or any other sort of engineer and more like motivating commercial artists with less pretense, who were never led to believe they were meant for something greater. Dissatisfaction from programmers grows in much the same way it grows in commercial artists as well, though they programmer is less likely to specifically identify his or her complaint and the artist is more likely to complain about having sold his or her soul.

Common responses to criticism of work among programmers align more with those among artists than those among engineers. Again, I learned this from a managerial perspective.

The most important advice that may be given to starting artists (excluding all the low-hanging fruit advice that is best for everyone in general, of course) isn't about discovering your own inner talent or anything similar, instead it is about discipline: "Ideas are not swords you can brandish about in triumph. What matters most is the Sit Down, Shut Up And Get It Done. Only there will you find the true steel for your craft. Only there, will you know if you are worth the words out of your mouth."

Comment author: Swimmer963 05 July 2012 08:16:24PM 3 points [-]

its hard to tell if talent or effort is more crucial for programming.

I would suggest talking to some programmers.

My intuition is that there's something of innate talent involved in programming, so that you can divide people into two populations: those whose brain makeup causes them to find programming intuitive and fascinating and cool, and those to whom it just doesn't make sense. If you're considering it as a career, presumably you fall into the first category. Beyond that, I would guess that conscientiousness is the biggest predictor–my one-semester programming elective was enough to show me that it's really time-consuming.

But I'm not a programmer by specialty. An unusual percentage of LWers are, though, so maybe someone can give you advice?

Comment author: Nornagest 05 July 2012 08:55:52PM *  4 points [-]

"The Camel Has Two Humps", which IIRC has been linked here before, does purport to find a bimodal distribution between people who can and can't program. I'm not at all sure if that has anything to do with inborn talent, though, at least beyond basic general intelligence.

At various points in my career I've found reasons to teach people programming skills, and my n=1 impression is that the ability to internalize basic programming has little to do with personality (though conscientiousness helps, and I suspect openness to experience might too) and a lot to do with the student's level of comfort with mathematical thinking. Not necessarily advanced math (you don't need anything more complicated than algebra to program except in specialized domains), but you do need to be very comfortable with a certain level of abstraction. I suspect that might have more to do with the distribution in the linked paper than the "geek gene" concepts I've heard tossed around elsewhere: at the level of the math prerequisites for CS 1 it's still possible to do well by solving problems mechanistically without a good grasp of the abstractions involved, but that won't cut it in computer science. And it'd probably be difficult to teach that in a semester.

Comment author: gwern 05 July 2012 11:52:16PM 5 points [-]
Comment author: ShardPhoenix 06 July 2012 03:33:53AM *  1 point [-]

The thing that wasn't replicated was their attempt at a predictive test of the distribution (based on a particular explanation they thought applied), not the existence of the distribution itself, which is something that was observed in grade patterns in CS compared to other subjects (though I don't know how rigorously established it is).

Comment author: gwern 06 July 2012 03:43:02AM 1 point [-]

Isn't the predictive part the interesting thing? I wasn't aware that bimodal grade distributions were unique to CS.

Comment author: ShardPhoenix 06 July 2012 05:15:00AM 0 points [-]

Well, their original paper claimed that (eg) math grades are typically a bell curve, whereas CS grades are typically bimodal (with examples from one university). But again, I'm not sure if this is something that's been rigorously demonstrated.

Comment author: Nornagest 06 July 2012 12:50:00AM 0 points [-]

Good to know. I thought it had a bit of a questionable odor to it, but I wasn't able to find any replications in the brief time I spent looking into it.

Comment author: Swimmer963 05 July 2012 09:33:40PM 0 points [-]

I don't think it has much to do with personality either, except, like you said, willingness to work hard (especially if you're someone who starts out finding it very difficult.) But I think that a lot of people, even people who can work up to the level of calculus in math, go at it with the mindset of "memorize that Formula X gives Answer Y" instead of trying to understand how and why Formula X relates to the underlying structure of the problem so that it's obvious that it should give answer Y, but gives Answer Z in a different context... You can get by with memorizing formulas in math classes, at least the way they're currently taught and tested. It's a lot harder to get by with that habit that when you're programming.

(On the whole, the people I've known whose minds appear to work like this aren't noticeably "lower" intelligence, however you define that. They just don't think of math as something where they should be applying the analytic part of their mind.)

Comment author: DaFranker 05 July 2012 08:53:31PM *  1 point [-]

If you define "talent" as a product of your current ability to produce and visualize mental models of complex systems, especially "from nothing", then it is the most defining factor for the higher maximum awesomeness of programs you can code at present.

This "talent" can be enhanced and self-improved through effort, however, in a very similar manner to making oneself more "luminous".

Comment author: brilee 06 July 2012 04:23:21AM *  7 points [-]

The concepts of positive and negative selection are not quite well defined in your essay, I think.

Imagine that you have one test, with a gaussian distribution of outcomes. Let's arbitrarily set a threshold, and if people are above this threshold, they have passed this test. Call the sets of passing A and not passing ~A

Would you call this a positive or negative selection? It is neither, in my opinion.

Now, imagine you have two tests, A and B.

A positive test is one where A U B are selected. A negative test is one where ~(~A ^ ~B) are selected.

In other words - the operative difference between positive and negative selection is OR vs. AND.

Comment author: torekp 06 July 2012 11:35:32PM 1 point [-]

Nice step toward clarity, but the leading example contrasts a single-test positive selection rule versus a multiple-test negative selection rule. I think it's worth a try to capture that contrast.

Here's a try. A selection process winnows down a pool of applicants (let's call them) to a pool of winners. A step in a selection process, where the steps are applied in a given sequence, is "absolutely negative" if it removes less than half the remaining pool. A step in one process is "more negative" than another step (in the same or another process) if it accepts a larger fraction of the pool.

Comment author: CronoDAS 06 July 2012 04:56:18PM *  17 points [-]

I asked my father to read this and give his thoughts.

He says that positive selection only works well when you have a very good idea what you need to select for. If you're sending an athlete to the Olympics but the event he'll have to compete in will be chosen at random, you can't just choose the one with the best time on the 800 meter dash, because the event might end up being something like archery, fencing, or weightlifting. And you certainly wouldn't want to send a non-swimmer. If you need a generalist, seeing how well someone does at jumping through a wide variety of arbitrary hoops might really be the best test you can practically implement.

(Now I'm wondering just how good or bad the 800 meter dash actually is at predicting levels of success at unrelated sports. For example, could you tell the difference between an NHL-quality ice hockey player and one that plays on a minor league team just by looking at their times on the 800 meter dash?)

Comment author: DaFranker 06 July 2012 05:42:36PM 8 points [-]

Assuming a significantly large distribution of athletes sent by other rational managers, where all athletes are bound to the same rules of random event selection, I would still send the best possible specialist in a single discipline in this case, because without certainty that all other rational managers know certainly that some generalists will be better in everything than other generalists and that each one is confident that theirs is best, I conclude that some of them attempt a gamble of probabilities and send a specialist, and thus I also send a specialist to maximize my chances of winning.

After all, there are higher chances of the event being my athlete's specialty than there are chances of every single other athlete being less good at it if I pick a generalist, unless the number of possible events is large enough to outweigh the number of athletes. Throw in irrational managers and the possibility of other managers having information unavailable to you, and your father's argument seems very weak.

Now, of course, I'm probably attacking something that wasn't meant to be a strong defensible argument. However, I feel very strongly about the point that negative selection is wrong in many contexts it is currently used in (which I support), as well as the point that positive selection is so difficult and utterly impractical in so many cases (which I want to pound into tiny bits of forgotten wrongness).

I'm not sure where I'm going with this, however. I strongly agree with the article's statements, but my attempts to formulate any further useful thought seem to come up short.

Comment author: CronoDAS 06 July 2012 06:12:57PM *  5 points [-]

Well, the sports analogy was my own interpretation of what he said.

Game theory question time: you and N other players are playing a dice rolling game. Each player has the choice of rolling a single twenty-sided die, or rolling five four-sided dice. The player with the highest total wins. (Ties are broken by eliminating all non-tying players and then playing again.) Now, rolling 5d4 has an expected score of 12.5 and rolling 1d20 has an expected score of 10.5, so when N=2, it's obviously better to roll 5d4. However, when N becomes sufficiently large, someone is going to roll a 20, so it's better to pick the 20-sided die, which gives you a 1 in 20 chance of rolling a 20 instead of a 1 in 1024 chance of getting five 4s. For exactly what value of N does it become better?

Edit: Fixed stupid math mistakes. That'll teach me to post after staying up all night!

Comment author: [deleted] 06 July 2012 07:46:43PM 4 points [-]

<nitpick>

rolling 1d20 has an expected score of 10

10.5

</nitpick>

Comment author: CronoDAS 06 July 2012 10:32:59PM 1 point [-]

Fixed, thanks.

Comment author: JGWeissman 06 July 2012 06:21:43PM 4 points [-]

1 in 256 chance of getting five 4s

4^5 = 2^10 = 1024

Comment author: CronoDAS 06 July 2012 10:32:51PM 1 point [-]

Fixed, thanks.

Comment author: DaFranker 06 July 2012 09:47:47PM *  3 points [-]

Insightful question, if you ask me, though solving for N feels a lot more like a straight up actuary-level math problem than Game Theory to me. My maths above basic calculus is generally foggy, so I'd appreciate any corrections or nitpicks someone more fluent here might have.

Essentially, you have to solve when (odds of having highest result when rolling d20) >= (odds of having highest result when rolling 5d4). To simplify, let's assume that all players are perfectly rational, and thus at N and higher will all roll 1d20. This still leaves you the problem of calculating N's odds of rolling higher than you for both rolls, which is a simpler reformulation of the above parentheses.

For any roll result Y, there is (y/20)^N probability that you "win" here, assuming ties count as wins (or at least are preferable to losses). This means that with N=1 (you're playing against one other person), you will win 52.5% of the time (and so will your opponent, because that 2.5% is for ties) when rolling 1d20.

Your odds of winning naturally decrease if you roll 1d20 such that for N=2 you have 35.875% chances of winning, and so on in a proportional manner since the odds are always even for everyone.

Where it gets more interesting is when you are playing an unfair game where you have to equate your total odds of winning when playing 1d20 vs d20s to those when playing 5d4 vs d20s. Since the math here is kind of foggy and hard to combine into one big formula, I've thrown the data at a spreadsheet (to calculate the sum of the odds of any N rolling higher than you for each roll Y multiplied by your odds of obtaining Y), and it turns out that at N=3 the 5d4 roll dips just below the odds of winning with 1d20 by about 0.2%.

However, if we want to compute for xDf die for N, with K possible ways to roll (which was 2 here), then the math yet eludes me. I've figured it out or been told what it was several times, but I just can't seem to ever memorize this when I can only barely remember integration anyway when I don't use it.

Edit: For those curious, here's the spreadsheet mentioned above with all the raw data and brute-force formulas.

Comment author: CronoDAS 07 July 2012 01:33:48AM 1 point [-]

Your analysis also assumes there's no difference between second place and last place.

Comment author: Viliam_Bur 07 July 2012 03:27:54PM 5 points [-]

Yes, the reward system is very important in choosing the right strategy. If the first place gives you gold, and all other places give you nothing, use positive selection. If the last places gives you problem, and all other places give you nothing, use negative selected. Other point of view: if being average is good, play safe by using negative selection; if being average is bad, aim for greatness (and accept a certain risk of failure) by using positive selection.

So the question is what exactly do we want in elite colleges or academia (examples from the article)? I guess for elite colleges it is better to play safe. If your students are above average and everyone knows it, they don't have to be exceptional -- your diploma will help them get a decent job, which is why they pay you. A few bad apples could ruin your marketing. With academia, for an average university it is probably better to have "safe" professors who do their jobs, get grants, and don't cause scandals; even if the price is having less Nobel-price winners.

Comment author: DaFranker 07 July 2012 03:06:51AM 0 points [-]

Yes, that it does, or at least it assumes that the difference is trivial within this decision scheme and the expected utility returns of a specialist are higher than the expected utility of a generalist even when taking second place into account.

Comment author: Sniffnoy 06 July 2012 10:13:42PM 3 points [-]

I don't think the right way to do this is not either positive or negative selection (those terms really suggest a false dichotomy, don't they?). As has been pointed out elsewhere, what's here being called "positive" is really "or", and what's here being called "negative" is really "and".

But there are lots more ways to combine the data into a single number then just "apply a cutoff to each one, and then apply some operation to the resulting booleans". The appropriate sort of selection is not positive or negative, but rather, whatever will be used in the actual competition. (And if it's unknown, apply expected utility, etc.)

Comment author: Vaniver 06 July 2012 06:41:19PM 0 points [-]

Now I'm wondering just how good or bad the 800 meter dash actually is at predicting levels of success at unrelated sports. For example, could you tell the difference between an NHL-quality ice hockey player and one that plays on a minor league team just by looking at their times on the 800 meter dash?

This (among other prior information) suggests to me that extreme levels of performance at different tests are probably negatively correlated, but I would not be surprised if there were events out there where extreme levels of performance on other tests are correlated with better (but not extreme) levels of performance on that test.

Comment author: [deleted] 06 July 2012 07:51:13PM 10 points [-]

Unless you sample at random from the whole population, that's a Berkson's fallacy.

Comment author: komponisto 06 July 2012 09:08:01PM 2 points [-]

Berkson's fallacy.

Upvoted for this link. I wish I had known this term back in the Amanda Knox days -- this fallacy (or rather, the reverse of it -- failing to take into account conditional dependence of a priori independent events) is a version of the main probability-theoretic error of that case.

Comment author: shminux 06 July 2012 10:30:09PM *  4 points [-]

This fallacy may also explain why people tend to assign 50% percent probability to the odds of the second child being a boy in the classic puzzle.

Comment author: Vaniver 06 July 2012 10:55:57PM *  1 point [-]

Thanks for the link! I think a careful statement of my claim avoids that fallacy.

The claimed data:

  1. Body shape is relevant to performance in particular events (say, the 800 meter dash)- for example, Michael Phelps is shaped well for swimming.
  2. Extreme performance in that event will generally require a body shape optimized for that event.
  3. Extreme performance in different events correspond to different body shapes.

Stated carelessly, it seems likely to me that if I know you're an Olympic-level athlete, I'll have some estimate that you can play Olympic-level basketball; but then when you tell me that you're a Olympic-level wrestler, I can lower my probability estimate that you would be able to play Olympic-level basketball.

But if I condition on knowing you're an Olympic athlete, and then try to drop that condition without being careful, then I can get into trouble (this fallacy, specifically). So instead, let's start off with some (really low) probability that a person chosen uniformly at random can play Olympic-level basketball, and then update on the knowledge that they're an Olympic wrestler- we should increase our estimate based on their general level of athleticism, and then decrease our estimate based on their probable body shape. I think the effect of the body shape will be stronger than the effect of general athleticism, and so they will actually be negatively correlated.

I think my last statement in the grandparent- that extreme levels of performance on a specific test should correlate with better performance on some 'general athleticism' test- is true when you compare extreme individuals to random individuals, but less true (or perhaps not true) when comparing extreme individuals to good individuals. The NHL ice hockey player is probably not much more 'athletic' than a minor league hockey player, but probably is more 'hockey-shaped.'

But now that I've stated that last paragraph, I'm thinking of counterevidence- like the famous birth month effect for Canadian hockey players. Early training appears to have a huge impact on whether or not someone hits extreme levels of performance, but I think both NHL players and minor league players started early. So maybe it is biological talent that differentiates many of them. Hmm. I should probably stop speculating about sports.

Comment author: CronoDAS 07 July 2012 01:32:03AM *  0 points [-]

If you put an Olympic-level wrestler on a college (American) football team, how well would they do? Michael Jordan did tolerably well during his year as a minor league baseball player.

Comment author: Desrtopa 07 July 2012 01:18:06PM 0 points [-]

Tolerably well for minor league, but remember that his father had envisioned him as a major league baseball player, so presumably he'd practiced and done well at the sport when he was younger. There are probably selection effects on Michael Jordan in particular to be good at baseball out of the set of NBA players; most never try to transition to baseball at all.

Comment author: Viliam_Bur 05 July 2012 11:03:41AM 6 points [-]

Grades in high school are already like this. To get the best grade from a subject, you need to be good, but not exceptional. So being good but not exceptional in everything brings you the highest possible score.

If instead you are exceptional in a subject or two, and average in a few unrelated subjects, it gives you lower total score, and if the university cared about your grade average, you would have problems getting there, especially if many people with the highest total score competed with you.

Comment author: JonathanK 06 July 2012 09:39:58PM 5 points [-]

Since most production functions are quasiconcave over inputs, negative selection is a cheap method of increasing expected return. You lose some outliers and also people who would be good in those rare domains with quasiconvex production functions, but our system is optimized for the average case.

In the college admissions example, a top school wants to admit undergraduates likely to become successful doctors/lawyers/businesspeople and alumni donors, not gamble that the smart kid with a few Bs in high school is going to revolutionize a scientific field in 15 years. Even if they did, their undergraduate institution would be only the third most important on their CV, after their current institution and where they got their PhD.

This is a good example of individual selection being suboptimal from a group perspective. Each top school would prefer that some other top school gamble on said smart kid, and then if they have the chops for research they can try to grab them when they apply to graduate school or go on the academic job market. Positive selection on undergraduates is just not a smart strategy from an individual institution's perspective since most undergraduates will be going into more conventional fields.

Comment author: Jay_Schweikert 05 July 2012 08:06:16PM 5 points [-]

Students at Yale are, for the most part, all strikingly similar – same socioeconomic class, same interests, same pursuits, same life goals, even the same style of dress.

Can I ask on what basis you're drawing this conclusion? I agree with the bulk of what you said about overuse of negative selection, but I challenge the idea that it's producing cookie-cutter student bodies at elite universities. Having attended Yale as an undergrad, your claim strikes me as incorrect, as the Yale student body seemed to me more diverse on all five of those categories than any other (non-online) community I've been a part of -- certainly more so than my high school, my law school, or my current professional environment. That's just my anecdotal experience, so I'm open to the idea that I'm mistaken if you have some more formal analysis to back this up. But this statement jumped out at me as questionable, so I wanted to see where you were getting it from.

Comment author: Vladimir_Nesov 05 July 2012 09:47:09PM 10 points [-]

He also attended Yale...

Comment author: gwern 06 July 2012 02:24:33AM 3 points [-]

I'm reminded of a quip from a former Yale professor:

A friend who teaches at the University of Connecticut once complained to me that his students don’t think for themselves. Well, I said, Yale students think for themselves, but only because they know we want them to.

As for hard facts and statistics, I imagine Charles Murray's recent book or Hayes's new book Twilight of the Elites (both of which I have heard good things about) might provide some.

Comment author: Dreaded_Anomaly 06 July 2012 05:34:48PM 6 points [-]

I've just realized that I have been treating dating as a negative selection process. This might explain the lack of success.

Comment author: pnrjulius 06 July 2012 10:49:21PM 3 points [-]

Well, you want some negative selection: Choose dating partners from among the set who are unlikely to steal your money, assault you, or otherwise ruin your life.

This is especially true for women, for whom the risk of being raped is considerably higher and obviously worth negative selecting against.

Comment author: Desrtopa 07 July 2012 02:00:41PM 0 points [-]

That carries the assumption that the qualities you're positively selecting for don't have a strong negative correlation with the ones you're trying to select against. I don't think it's hard to lay out a few basic "are" qualities that imply "are not" for "violent, thief, etc."

Comment author: Viliam_Bur 07 July 2012 03:54:06PM *  8 points [-]

If A means not B, then selecting for A is the same thing as selecting against B.

If A means "with probability 90% not B", then if B is a serious problem, it is worth checking both A and not B. Maybe even checking not B first, to avoid halo effect from A.

In my experience, some people treat dating as a negative selection process with thousand requirements that no one passes, because thousand criteria are simply too much. (Assuming independent results, even with probability 99% of passing each test, less than one person in 20 000 passes all thousand criteria. In real life, the criteria are often positively correlated, but on the other hand the probability is way less than 99%.) And those people usually defend it by taking each criterium out of the context and saying: "What's wrong about wanting my boyfriend/girlfriend to be interested in opera/programming?" Well, nothing wrong per se, but if you have thousand criteria like this, good luck finding a person who fulfills them all (and is also interested in you).

The solution is to separate those criteria into two groups: "must have" and "nice to have". (And if nine hundred of the thousand criteria are in the first group, you are doing it wrong.) First, filter people by the "must have" criteria. What remains is your dating pool. Some of those will be never interested in you, but you will find that out by trying. Now use the "nice to have" criteria for a utility function, and go seduce someone with a high utility. (And as a parallel process, try to increase your market value.) At the end, you may find someone who has all "must have" and some of the "nice to have" traits; and you may be happy with them.

Comment author: Tuesday_Next 03 August 2012 12:10:13AM 1 point [-]

If A means not B, then selecting for A is only the same thing as selecting against B IF A doesn't also mean other things, besides not B.

In the dating example, a (straight) woman might employ positive selection to choose men who are particularly decent people. This would also have the effect of weeding out thieves and rapists (assuming that the woman in question can assess a man's decency with sufficient accuracy), but the quality of "being a decent person" doesn't only mean one isn't a thief and a rapist; it's more wide-ranging than that.

Comment author: Vaniver 07 July 2012 10:11:29PM 1 point [-]

This might explain the lack of success.

Doubtful. My romantic excitement, as best as I can tell, follows a positive selection process and it is highly inconvenient. (Basically, my dating history looks like I sat down at 16, compiled a list of people who fit a particular description, sorted them along that axis, and then have tried to date them.) Like Viliam_Bur points out, you don't want to have 900 criteria which can disqualify someone- but you do want to have a reasonable number of thresholds that you don't go below. At some point, the amount you dislike the other person will be determined by their lowest stat that you care about, and it would be nice to not have to deal with a lot of dislike.

Comment author: fburnaby 08 July 2012 08:19:31PM *  0 points [-]

It seems that Vaniver and pnrjulius have assumed that you're having trouble picking good dates. If, instead, you are worried about getting picked (or accepted) for dates, then maybe you're on to something. I'd be interested in knowing whether the majority of people accept dates based on a positive or a negative selection process. It may need to be broken down by gender.

(I have a hypothesis that I won't share yet, in case it influences results)

Comment author: Dreaded_Anomaly 08 July 2012 11:23:30PM 1 point [-]

It seems that Vaniver and pnrjulius have assumed that you're having trouble picking good dates.

Viliam_Bur describes my thought process correctly.

I'm faltering on the first step, finding a woman whom I would be interested in dating. I think part of this is due to what I now recognize to be too many criteria ruling out people who might otherwise be appealing. (I've certainly had people tell me I'm too picky before, but it took a comparison to undergraduate admissions for the underlying nature of the problem to become apparent.)

Worrying about getting picked or accepted is a different step entirely.

Comment author: fburnaby 09 July 2012 11:06:54AM 0 points [-]

That definitely makes it clear what your intention is.

Comment author: fburnaby 08 July 2012 08:20:33PM 0 points [-]

I'm male and (I think) I tend to apply negative selection when deciding.

Comment author: johnlawrenceaspden 06 July 2012 12:35:39PM 2 points [-]

I've never understood the reason for giving grades A-E or fail, like we do for O and A levels, or I:II:III:fail, like we do for degrees.

My father's O-levels gave a percentile ranking, so he was e.g. in the 83rd percentile in the country for history.

So we must have changed over at some point. Does anyone know why? It's always looked like throwing information away to me, and it's also unfair to people on the grade boundaries.

Of course this may be motivated thinking on my part, I'd much rather have had a string of 100s for my exams than a string of As, and I'd much prefer to have got a 75 for my degree than a II (which covered percentiles 25-75) !!

Comment author: pnrjulius 06 July 2012 10:51:05PM 0 points [-]

I think a lot of people don't like using percentiles because they are zero-sum: Exactly 25% of the class is in the top 25%, regardless of whether everyone in the class is brilliant or everyone in the class is an idiot.

Comment author: Viliam_Bur 07 July 2012 04:08:00PM *  0 points [-]

But on the opposite side of the spectrum is saying: "Everyone is so smart because they can read and write, and thousand years ago most people couldn't do this!" (Strawman example, I know.)

Generally I would prefer to have a list (tree, directed acyclic graph...?) of all human knowledge, and give everyone a report saying: "This person understands these parts." But over time, the list/tree is growing. Of course it is OK to know a smaller part of total human knowledge, because the population is growing; but still, you need to know more than your ancestors (if your computer skills are the same as your grandma's, then she is a hero and you are a loser); on the other hand some knowledge becomes obsolete.

I think a percentile across the whole country would be a good measure for comparing individual students or schools. And it would be nice to also calculate long-term changes to know whether the country as a whole is improving.

Comment author: DavidAgain 12 July 2012 06:26:34AM 1 point [-]

Of course, you could show percentage scores in the tests rather than where you sit in the country. That means that it should be consistent over time, although I agree that in a decent sized national subject it's probably fine the other way.

My main objection to giving percentiles relates to the OP's concern that there's no such thing as a 'very good' A, At least with UK school exams, I think that getting 100% in most subjects tests for conscientiousness and not making silly errors at best and being well-trained in the exam system at worst. I am pretty sure that if percentages were public I'd have had to get better marks to get into uni, but also that in making sure I did so I would not have been using my time usefully.

What I think would be far preferable to a 'who managed not to screw up a single question' model of getting better than an A would be an extension paper that was genuinely challenging and couldn't be straightforwardly taught.

Comment author: MixedNuts 08 July 2012 09:43:25AM 0 points [-]

Boy, are you setting a high bar for Erna Hoover's grandkids.

Comment author: Vaniver 05 July 2012 02:37:25AM 2 points [-]

There’s no one way you can do really, really well, and thereby be admitted to Harvard.

You don't think Jack Andraka will get admitted to Harvard because he did one thing really, really well?

Comment author: komponisto 05 July 2012 02:48:59AM 1 point [-]

Not if his high-school GPA is 2.0.

Comment author: Vaniver 05 July 2012 03:14:40AM 2 points [-]

I think more highly of Harvard's admission department than that. At the very least, Andraka has shown himself capable of finding a champion within the system through persistence.

Comment author: AlexMennen 05 July 2012 04:00:49AM 18 points [-]

I got into UC Berkeley with a high school GPA of 2.9 by talking about math with professors. This strategy failed everywhere else, and would have failed at Berkeley if I hadn't been lucky enough to find a professor stubborn enough to argue with the admissions office again after they ignored him the first time. On the other hand, my accomplishments are not even close to as impressive as Andraka's, so he might have an easier time with this strategy even with a worse GPA.

Anyway, if you've done anything impressive, finding a champion within the system is easy. Andraka had a hard time with that step because he was trying to get support before doing something cool rather than after. Now, the vast majority of biology professors would gladly stand up for him to their institution's admissions department. But this strategy requires persistence on the part of the champion, as well as the applicant.

Comment author: komponisto 05 July 2012 03:09:07PM 5 points [-]

Anyway, if you've done anything impressive, finding a champion within the system is easy.

It's not impossible, but "easy" is an overstatement.

One of the most disappointing discoveries of my life was the existence of professors -- even math professors, alas -- who think like high-school teachers or college admissions officers. They not only exist, but exist in large enough numbers that one will actually run into them. The good guys also exist, but they don't dominate the way I thought they did. This realization caused me to change my view of academia.

Now, the vast majority of biology professors would gladly stand up for him to their institution's admissions department.

Again, I think that's an overestimate. Maybe half of them would, but the "vast majority"? You would get a substantial number who, while grudgingly admitting the impressiveness of his accomplishment, would seek to rationalize the traditional status structure by making excuses about his not being "ready" or "a good fit", etc.

Comment author: AlexMennen 05 July 2012 10:03:31PM 2 points [-]

When I did it, it wasn't very difficult. Maybe about a third of the professors I talked to agreed to talk to their admissions department on my behalf, which is remarkably high considering that I had not received any official recognition for my work, I had some difficulty explaining it coherently, it was really only slightly impressive, and I'm a bit short on social skills, which are useful for impressing people (I had a lot of really awkward meetings with professors). Andraka at the very least has a huge advantage over me in the first and third of those problems, which should give him a sizable majority. If he's solid in the second and forth problems as well, I would expect him to get an overwhelming majority.

Comment author: komponisto 05 July 2012 03:33:08AM 5 points [-]

I think more highly of Harvard's admission department than that.

Any particular reason?

Comment author: Vaniver 05 July 2012 03:59:28PM 1 point [-]

My primary reason is I expect that they'll have the best one that money can buy- and I suspect that schools like Harvard explicitly look for a blend of students (since a system that gives them only Senator's sons or a system that gives them only brilliant kids will lower the value of Harvard to both, but a system that gives them, say, 30% Senator's sons and 70% brilliant kids will do better).

Comment author: komponisto 05 July 2012 04:31:33PM *  5 points [-]

My primary reason is I expect that they'll have the best one that money can buy

I think this is wrong. To be sure, they could have that if they wanted, but that doesn't seem to be where they actually spend their (considerable) money. From the book A is for Admission: An Insider's Guide to Getting Into the Ivy League and Other Top Colleges:

I hope by now the point is clear: For the most part, Ivy League hotshots are not the ones reading your application. You will note the conspicuous absence of Rhodes scholars or well-known educators on admissions staffs...As my former colleague from the Putney School in Putney, Vermont, and former Brown admissions officer Harry Bauld writes in his hilarious book on college essays, "This is your audience. Study them well. Not exactly the Nobel Prize panel."

\

and I suspect that schools like Harvard explicitly look for a blend of students

Yes, but they have a formula for achieving the blend they seek, and that formula is going to filter out people with low GPAs and such.

Comment author: Vaniver 05 July 2012 04:41:44PM 2 points [-]

I know that the admissions staff are generally mediocre students who went to that school. But I would expect their system to notice things like "Intel Science Fair winner" and have that trump any GPA signal.

(More reasonably, I think they have a file of names called "admit these people", which it checks applicants against (and if they're on the list, sends it to a human to verify), and Andraka's win was publicized widely enough that he probably made it onto the list, and if someone is doing their job well they routinely import the list of winners from things like the Intel Science Fair into that list.)

Comment author: magfrump 05 July 2012 11:19:36PM 3 points [-]

(More reasonably, I think they have a file of names called "admit these people", which it checks applicants against (and if they're on the list, sends it to a human to verify), and Andraka's win was publicized widely enough that he probably made it onto the list, and if someone is doing their job well they routinely import the list of winners from things like the Intel Science Fair into that list.)

I'm fairly confident that this is a thing that actually exists, because of the associated prestige. Universities would get this if they were optimizing for status, without optimizing for learning at all.

Comment author: komponisto 07 July 2012 05:13:51AM 2 points [-]

However, you also have to consider marginal payoff relative to the cost. Most Science Fair winners will also score high according to the standard formula (involving GPA et cetera); any additional prestige the institution would gain by also admitting the very few who don't probably wouldn't be worth the cost of having such a separate system.

Comment author: magfrump 07 July 2012 07:53:44PM 0 points [-]

Unrelated to my other points: When in your experience have universities acted efficiently, as opposed to just "do things that sound like they'll increase status"?

Comment author: magfrump 07 July 2012 07:52:59PM 0 points [-]

While in some senses I agree, the whole process of admissions just consists of people putting stamps on paper. If one of those people recognizes someone from a news article and just says "hey let's stamp this" it doesn't actually require more bureaucracy. Since all your processes are run by humans it doesn't actually cost anything to add human judgment to your system.

For example, I would be EXTREMELY SURPRISED if there was a computer program that STOPPED a university from admitting someone if they had too low a GPA. It's just that the computer program wouldn't present them to be considered in the first place unless they looked.

In terms of practical tests, I propose that if we look up the set of Intel Science Fair winners, see if there's information about their GPAs, and then look at what universities they got into, I hypothesize that if there are any with GPAs below, say, 3.7, they will still get in to high end universities that normally would only accept students with 4.0s (Stanford, MIT, Harvard, Carnegie Mellon, Johns Hopkins come to mind). I agree that it's unlikely that you'll find any with recorded GPAs below say 3.0, so the question may be purely theoretical anyway.

Comment author: Eliezer_Yudkowsky 05 July 2012 06:18:26PM 5 points [-]

If you can find a way to settle the bet, I bet they don't do that. Universities would look extremely different if they optimized for learning in even the most basic ways. This is the should-universe you're talking about, not the is-universe.

Comment author: [deleted] 07 July 2012 06:24:42AM 8 points [-]

I have a dean of admissions at a large university in my nuclear family. Eliezer is right, there's no list like this.

But on the other hand, "Intel Science Fair winner" will PROBABLY attract the attention of the admissions committee. It's basically up to the applicant to craft a good applications package (including essay and letters of recommendation) that will capitalize on their amazing, singular strength, and throw weaknesses like GPA into shadow. If the applicant can't do this, they won't be admitted.

It's a Bardic Conspiracy problem, really. It's a matter of storytelling and presentation.

Comment author: Eliezer_Yudkowsky 07 July 2012 07:47:20AM 13 points [-]

I note that it also makes no sense to filter excellent scientists who aren't good writers or who take a long time to write (e.g. PhD dissertation test). If you can do the research, someone else should be able to specialize in writing for you.

It's remarkable how many barrier-to-entry professions revolve around the denial of professional specialization. A surgeon can't just be someone of moderate intelligence and exceptional dexterity who studies and practices one key surgery, no, they need 11 years of medical school that they'll mostly never use. A scientist is forced to write. And so on.

Comment author: Ezekiel 05 July 2012 06:29:01PM 3 points [-]

Out of genuine curiosity, how do you know that? I thought you never went to university.

Comment author: DaFranker 05 July 2012 09:24:17PM 1 point [-]

Personal experience, most likely. What little I've seen / know of his knowledge indicates in-depth mastery of multiple topics that would each have taken five or more years of university courses to learn.

Having learned them all from university courses without special exception being made (that is, taking full-term courses without any skipping of courses or taking more than six courses per term) is highly improbable.

Many of my thought experiments into forming universities or educational institutions in general more geared towards optimized learning (e.g. open-learning systems where each student is at different levels in different subjects, and takes tests when milestones are reached rather than at specific predefined dates) seem to strongly indicate that while many of them would be much better for making more intelligent individuals or letting people learn much faster, the optimal utility-maximizing situation for the "Institutional Governing Body" is the current system. In other words, the individuals in positions of power to change the institutions have much more to gain (at least in the short term on their personal utility scales) in maintaining the current system. All my calculations, estimates and observations so far have consistently been in agreement with this statement, though I suspect a great deal of personal bias is at work here.

Comment author: Vaniver 05 July 2012 07:14:17PM 0 points [-]

If you can find a way to settle the bet, I bet they don't do that.

The closest thing I can think of is contacting people in the admissions department, but I can't think of a cheap way to induce them to answer truthfully.

I'm also willing to consider humans part of the 'system', and so having that 'file' be "Bob recognized this applicant's name" would be enough for my purposes. But I don't know how much human attention their applicants get, and at what parts.

Comment author: hairyfigment 05 July 2012 06:25:35AM 4 points [-]

Yeah, I thought while reading the OP:

  • One could likely get into Harvard by winning a Nobel Prize, or being the guy from "Good Will Hunting".

  • This method seems neither guaranteed nor widely known.

Comment author: DanArmak 05 July 2012 06:34:01PM 4 points [-]

If you've already got a Nobel, what do you need to get into Harvard for?

Comment author: CronoDAS 06 July 2012 01:35:06AM 1 point [-]

Maybe you want to study something you're not already an expert in? (For example, Feynman spent some time taking graduate biology courses.)

Comment author: fubarobfusco 05 July 2012 11:53:07PM 0 points [-]

Why would he want to go to Harvard specifically? For a research career, he should be choosing specific programs and labs — not choosing on the basis of the broad general reputation of a university. For undergraduate, though, an institution with a culture friendly to weird people (and anyone doing that level of research at age 15 is gonna be pretty weird) might be more rewarding than a hothouse of Future World Leaders.

Comment author: Vaniver 06 July 2012 12:30:33AM 1 point [-]

"Should go to" and "will get admitted to" are different things. I think it likely that Andraka will go to Johns Hopkins, for obvious reasons.

My point is more that positive selection mechanisms do exist- they're just a small part of the general pipe, and people are trained away from hunting for them instead of just putting up with the trouble of negative selection.

Now, tommccabe's point still stands if the word "one" is the most significant word in that sentence. There are a host of ways to raise yourself to Harvard's attention and clear aside the barriers put up to keep the riff raff out- but they are non-obvious and (deliberate) chinks in the general system, rather than the primary way the system operates.

Comment author: komponisto 05 July 2012 02:12:34AM 2 points [-]

This is a very important point; thanks for cross-posting.

Comment author: shminux 05 July 2012 05:53:00AM *  5 points [-]

I agree with most of it, though the point about academia is a bit contrived.

True, there is a lot of negative selection before you get a cushy job the usual way, but you can certainly bypass quite a few obstacles if you are exceptionally good. For example, solve any of the open problems in math or physics, post a preprint on arxiv.org (well, you may need someone to vouch for you, but that's not really an issue) and you are all set.

Unfortunately, I cannot recall a single discovery in physics in the last half a century that was not made by someone who jumped through the usual hoops. I have met, however, an occasional person who learned grad school-level stuff on their own, but they did not manage to go any farther. My suspicion is therefore that all that negative selection in science, while annoying, does not do a lot of harm compared to potential alternatives. While it filters out some good people, it probably does not reject the very best, otherwise we would see an occasional example of someone making a significant discovery outside academia.

Comment author: Desrtopa 07 July 2012 02:24:41PM *  5 points [-]

Unfortunately, I cannot recall a single discovery in physics in the last half a century that was not made by someone who jumped through the usual hoops. I have met, however, an occasional person who learned grad school-level stuff on their own, but they did not manage to go any farther. My suspicion is therefore that all that negative selection in science, while annoying, does not do a lot of harm compared to potential alternatives.

But you don't get to observe any of the discoveries in physics that haven't been made. If a good university education is markedly better for learning physics than autodidactism, then the people who don't jump through the usual hoops will be inhibited by an inferior education and won't be in the position to make discoveries that a person who did jump through those hoops is.

If receiving a high level education and having access to university resources is effectively a precondition for making significant new discoveries in physics today, you would not expect to see people who did not go through the regular procedures making significant new discoveries in physics, even if the negative selection of academia filters out most of the best candidates.

Comment author: [deleted] 07 July 2012 04:07:57PM *  0 points [-]

If receiving a high level education and having access to university resources is effectively a precondition for making significant new discoveries in physics today, you would not expect to see people who did not go through the regular procedures making significant new discoveries in physics, even if the negative selection of academia filters out most of the best candidates.

But remember that these universities are trying their very, very best to select the best candidates, and the people doing the selecting tend to have a lot of experience and to have done a lot of thinking about how to select the best candidates in their field. For every old fuddy duddy who only selects people his antiquated views allow for, there's a vital young department head out there hunting for the radical new genius. Well funded departments often keep standing job positions open just for when that rare and exceptional person shows up. I don't think we have any reason to believe that academic departments are bad at finding talented people.

And given how willing people are to throw money and time and resources at scientific and mathematical talents, it seems fair to say that if someone wants to contribute to math or physics (and are capable of doing so), being an autodidact is an almost always an extremely bad strategy. The hoop jumping isn't really that hard or time consuming.

Comment author: Desrtopa 07 July 2012 05:11:43PM 1 point [-]

But remember that these universities are trying their very, very best to select the best candidates, and the people doing the selecting tend to have a lot of experience and to have done a lot of thinking about how to select the best candidates in their field.

I think this is simply false as a matter of fact. See this comment.

Do I personally think that the negative selection of academia filters out a lot of the best candidates? Not really; outstanding success in any discipline is heavily dependent on effort (which is not to say that it will not also require a lot of talent,) and I suspect that most people who're capable of making really top level contributions are able to put in enough effort that they can get through the necessary selection barriers, even ones dependent on skills in which they have low aptitude. But this is a very different matter from supposing that schools are actually good at selecting the best students; with sufficient effort and conscientiousness after all, one could get through moderate strength filters completely orthogonal to the skills one intends to specialize in.

I think that the people that the current negative selection system impacts most are probably not people with remarkable singularly focused genius, but people who could be pretty good specialists, who lack the aptitude to pass some filter orthogonal to their specialty.

Comment author: [deleted] 07 July 2012 05:25:47PM *  0 points [-]

I think this is simply false as a matter of fact. See this comment.

Ah, I was speaking in terms of finding talented researchers at the graduate and faculty level, not lvy league undergrad admissions. Your comments seem reasonable on the subject of undergrad admissions, which I agree are almost wholly negative selection filters. Do you think this also goes for graduate admissions? What should we anticipate observing if schools were bad at this kind of selection? What would we see if they were good at it?

EDIT: I'm working largely on the observation that getting a PhD in any field is really very easy. The major barrier seems to be interest. This doesn't go for all fields, of course. Law is a serious exception. But physics? Mathematics? I lack data here, but I'm skeptical that these are particularly closed academic fields.

Comment author: shminux 07 July 2012 09:19:16PM 2 points [-]

I'm working largely on the observation that getting a PhD in any field is really very easy.

For some definition of easy... As most grad students know, phdcomics is a documentary. The majority of grad students have breakdowns and burnouts as a matter of course. Most (70-75%) still finish, only to never set foot in academia again.

Comment author: [deleted] 08 July 2012 08:02:41PM 0 points [-]

The majority of grad students have breakdowns and burnouts as a matter of course.

I've never understood this, but I cannot deny that it's true.

Comment author: Luke_A_Somers 08 July 2012 08:28:44PM 2 points [-]

Then perhaps there's something about getting a PhD that you don't understand? And, given that, maybe you shouldn't make such sweeping statements about how easy it is?

Comment author: [deleted] 08 July 2012 08:31:14PM 0 points [-]

Perhaps you'd like to enlighten me? Do you understand why PhD students so often burn out? I'm a PhD student, and I don't get it. It's stressful, but not as stressful as a lot of other jobs.

Comment author: Luke_A_Somers 09 July 2012 01:37:49PM 1 point [-]

Some things off the top of my head.

Having to be far more independent a researcher than you ever had before.

The standards for your performance have risen similarly.

Projects that don't work, no matter what you do.

Projects that would have worked if you had done them right.

Having to deal with stupid university-political things.

Low pay, with long hours

... while you're listening to the biological clock ticking

... and it's very likely you'll have to move in a few years, so settling down will make things very tricky later on. The '2-body problem' is rough.

Possibility of having your result 'scooped' by someone

... who read your paper in review, held it up with BS, did a quick measurement to reproduce it, then got it published first.

Comment author: wedrifid 07 July 2012 10:07:07PM 0 points [-]

EDIT: I'm working largely on the observation that getting a PhD in any field is really very easy. The major barrier seems to be interest.

I'd contradict that. The entry requirements and task is utterly trivial and the intellectual and academic challenge isn't much of a big deal for anyone with average (or perhaps slightly above average) IQ but the motivational aspect is ridiculous.

Comment author: [deleted] 07 July 2012 11:59:49PM 0 points [-]

The entry requirements and task is utterly trivial and the intellectual and academic challenge isn't much of a big deal for anyone with average (or perhaps slightly above average) IQ but the motivational aspect is ridiculous.

Isn't that what I said?

Comment author: handoflixue 11 July 2012 09:11:33PM 0 points [-]

No, you said it was "really very easy", which implies that the motivation aspect is trivial. Something can be difficult FROM a motivational standpoint.

Comment author: Desrtopa 07 July 2012 05:58:17PM *  0 points [-]

Do you think this also goes for graduate admissions? What should we anticipate observing if schools were bad at this kind of selection? What would we see if they were good at it?

On the face of it, this strikes me as tough to judge, because if they're bad at it we should expect to see e. g., people whose abilities within their field are highly apparent to their professors and peers failing to get into programs which people who make a lesser impression on those who know them do get into, but it's hard to aggregate anecdotes like this to make strong generalizations.

If you could run two academic environments alongside each other, one of which was good at selecting for high aptitude and one of which wasn't, I expect it would be pretty easy to differentiate between them, but it's much harder without a basis for comparison.

I'm working largely on the observation that getting a PhD in any field is really very easy. The major barrier seems to be interest. This doesn't go for all fields, of course. Law is a serious exception. But physics? Mathematics? I lack data here, but I'm skeptical that these are particularly closed academic fields.

I've never gotten into a PhD program for either (I've never entered a PhD program period, for that matter,) but I don't share this impression and am curious as to how you formed it.

Even getting into a PhD program in things that we're inclined to regard as "soft" subjects, such as English, can be quite difficult, since there's so much competition. "Good enough" means "better than a whole lot of other people who're also sufficiently invested to apply to the same program."

Of course, "very easy" is relative, but I think most people would not agree with the assertion that getting a PhD is easy.

Comment author: [deleted] 07 July 2012 07:07:00PM 0 points [-]

I've never gotten into a PhD program for either (I've never entered a PhD program period, for that matter,) but I don't share this impression and am curious as to how you formed it.

It's not based on very much, I admit, beyond talking to some physics and math people I know about graduate school. It is a selection procedure, and it does take some significant work and talent to get in (as well as some luck), but it doesn't strike me as unreasonably difficult. All the physics and math PhD's I know seem genuinely (but not off the charts) talented. Many of them have problems finishing their work on time and writing well. Once you're in the program, getting out the door with a PhD isn't that hard, since literally everyone making the decision to graduate you wants you to graduate.

What I'm trying to say is that getting a PhD is hard in the way building a nine foot brick wall is hard. There are some basic skills involved, and then it's mostly just a lot of time and work. It's not hard like discovering a new proof in geometry is hard. And if you can do the latter kind of work, people are pretty inclined to cut you some slack on the former. My school recently hired a mathematician who earned her PhD at 24, six months after her BA.

Even getting into a PhD program in things that we're inclined to regard as "soft" subjects, such as English, can be quite difficult,

I'd say these are much, much more difficult to get into. Firstly because there are way more candidates (the negative selection pressures are a lot lower in these fields) and secondly, because it's not at all clear what talent in these fields looks like, so things are a lot more random.

Comment author: komponisto 07 July 2012 05:56:28PM 0 points [-]

I'm working largely on the observation that getting a PhD in any field is really very easy. The major barrier seems to be interest. This doesn't go for all fields, of course. Law is a serious exception. But physics? Mathematics? I lack data here, but I'm skeptical that these are particularly closed academic fields.

Yes, I think you're mistaken about this. Relatively speaking, you're correct -- fields like mathematics have less "academic snobbery" than law, humanities, and the like. The problem is that relatively isn't enough. There is still enough snobbery present that people with "blotches on their record" need a significant amount of good luck in order not to be filtered out.

Comment author: [deleted] 07 July 2012 06:57:21PM 0 points [-]

There is still enough snobbery present that people with "blotches on their record" need a significant amount of good luck in order not to be filtered out.

Well, the question is whether or not academic programs are good at selecting for talent (and, for the purposes of my point, I mean at the graduate level and above). So you may be right that it's hard for people with blotches on their record to do well, but does this make the selection process a bad one, considered on the basis of costs to benefits?

Comment author: komponisto 07 July 2012 08:11:02PM 1 point [-]

Yes: there are excessive numbers of both false negatives (talented people with blotches) and false positives (untalented people without blotches).

I'll make up an example (loosely based on some real cases I know about) to illustrate what I mean. Suppose a graduate admissions committee in a math or physics department is looking at two candidates. Candidate A is applying to transfer from another graduate program at another institution, where for some reason he was a total disaster, flunking courses left and right. On the other hand, his undergraduate record is near-perfect, and itself consists mostly of graduate-level courses. Candidate B, by contrast, is applying directly from undergraduate school, and has a reasonably good record, but with standard undergraduate coursework in the subject, nothing particularly beyond "grade level".

Now, of these two candidates, which do you think has a higher chance of admission? Neither of them is ideal, of course; but let us stipulate that exactly one of them is in fact admitted. Who do you think it was?

If you were trying to select for talent, it would be Candidate A every time. And sure enough, there are people in math/physics academia who would indeed pick Candidate A. (This is an improvement over high school, or humanities academia, where no one would.) However, a disturbingly large number will pick Candidate B. They will see Candidate A as "damaged goods", as "unworthy" -- as if a spot in a graduate program were a reward for "good citizenship", rather than a resource to be used for getting research problems solved.

Comment author: [deleted] 07 July 2012 08:23:35PM 0 points [-]

Yes: there are excessive numbers of both false negatives (talented people with blotches) and false positives (untalented people without blotches).

Why do you think this?

Who do you think it was?

So I haven't read your last paragraph yet. I guess if I were making the decision, I'd pick the undergraduate, unless the graduate transfer had some kind of story about what happened (a divorce, a death, something that would explain the failures). I think that's probably how most admissions committees would go too: talent isn't worth much if you need to be dragged to work every day. Now, on to your last paragraph...

(This is an improvement over high school, or humanities academia, where no one would.)

About humanities academia, this isn't at all true.

Comment author: komponisto 07 July 2012 09:17:16PM 0 points [-]

I guess if I were making the decision, I'd pick the undergraduate, unless the graduate transfer had some kind of story about what happened (a divorce, a death, something that would explain the failures).

Well, of course there's some kind of story that explains the failures, other than "he just wasn't good enough"; by assumption the guy wasn't let into the first graduate school off the street -- he had been a brilliant undergraduate. So something went wrong. The question is whether you should filter out a clearly capable person because something went wrong once. If you do, you're simply not optimizing for the right thing.

A rational deliberation on Candidate A would look like this: "This person obviously has issues, and he might not succeed here. But if he does, there's a good chance he'll succeed spectacularly -- at least, at a level above that of many students we're happy to call 'successful'. If he doesn't, well, some percentage of the blemish-free students we admit are going to fail anyway, so what's the difference?"

It is a fact of life that sometimes multiple iterations are required for something to work; but the school system does a poor job of accommodating this. Repeating a course as many times as necessary until you get an A is not something that the system encourages; if it were optimizing for the right things, it would be.

About humanities academia, this isn't at all true

Yes, that was a flippant, stereotyped exaggeration -- especially since "humanities" includes philosophy and linguistics, where the mentality is often very similar to math. Still, like any good caricature, it has some basis in reality. Generally speaking, the "softer" the subject -- the fewer objective measures of competence -- the greater the reliance on pure status games, i.e. academic snobbery. And of course my whole point here is that this snobbery is present even in "hard" fields.

Comment author: Viliam_Bur 07 July 2012 03:58:30PM 0 points [-]

If a good university education is markedly better for learning physics than autodidactism

This may be somewhat helped by having university education available online. (Although having the patience to see the online courses and do the exercises is also kind of a hoop. But at least it does not say you when you must do it.)

Comment author: komponisto 05 July 2012 04:11:13PM 8 points [-]

While [negative selection] filters out some good people, it probably does not reject the very best, otherwise we would see an occasional example of someone making a significant discovery outside academia.

I predict that we will indeed see this before too long, now that we have the internet; and it will thus turn out that some of the best people were being filtered out. Access to information and social support/reinforcement is a huge limiting factor.

And of course, if you're willing to look a century back instead of just a half-century, you find the salient example of Einstein -- who didn't even have the internet, but still managed to advance science from outside the "establishment" (which was a sizable apparatus in his time and place, just as it is in ours).

Comment author: DanArmak 05 July 2012 06:30:51PM 5 points [-]

Access to information and social support/reinforcement is a huge limiting factor.

Access to labs, equipment, technicians, funding is an even greater factor. Only mathematicians can really afford to work from home. (And now, computer scientists and computational-xxx people have joined them.)

Comment author: komponisto 05 July 2012 07:35:20PM 3 points [-]

Yes, all my predictions about people working at home should be interpreted to refer to fields in which that is physically possible.

(In fact, in these discussions I am pretty much always thinking specifically of mathematics, and possibly the most theoretical kinds of physics.)

Comment author: pnrjulius 06 July 2012 10:58:49PM 0 points [-]

It's not quite so dire. You can't do experiments from home usually, but you can interpret experiments from home thanks to Internet publication of results. So a lot of theoretical work in almost every field can be done from outside academia.

Comment author: DanArmak 07 July 2012 12:56:25PM 0 points [-]

Yes, but in most fields someone can't participate by only interpreting experiments from home. It's useful, but you can't build a career from it. Normally you really want to also be able to influence experiments in the lab to get the new data you want.

Comment author: shminux 05 July 2012 06:51:25PM 3 points [-]

I predict that we will indeed see this before too long

I am willing to bet that none of the high-profile open problems in physics, such as quantum measurement, high-temperature superconductivity, dark energy origins, extensions to the standard model etc., will receive a meaningful contribution from outside of people trained in academia, at least not in the next 10 years. The reason is that the cream-of-the-crop people who are able to advance the leading edge stand out enough to be recognized and integrated into the system.

And of course, if you're willing to look a century back instead of just a half-century, you find the salient example of Einstein

This is a myth. While he had trouble fitting in, he certainly did jump through most of the usual hoops.

Comment author: komponisto 05 July 2012 07:42:54PM *  4 points [-]

If a person is recognized and integrated into the system only after making a contribution, that counts as being "outside the system". E.g. Einstein, who didn't get his first academic position until 1908, three years after the annus mirabilis, which had occurred while he was a patent clerk. Indeed, he didn't even get his Ph.D. until the annus itself -- his thesis consisted of one of the famous papers!

Comment author: shminux 05 July 2012 08:18:13PM *  4 points [-]

He got his undergrad degree in 1900, so, inside the system. He was a lecturer (part time) between 1900 and 1902, though not at a high level. He did indeed develop his SR ideas while dealing with electromagnetic applications while working at the patent office (which paid the bills), though still in touch with other scientists, most notably Marcel Grossmann. His thesis was not related to relativity at all, not that it matters. All of his work on GR was inside the system. Not that he needed the system much by then.

So, the popular view of Einstein as an outsider is blown way out of proportion to reality.

Comment author: komponisto 05 July 2012 08:50:57PM 4 points [-]

Whatever the "popular" view, all that matters for my purposes is that Einstein was not employed by a university in 1905 when he developed SR, and thus was, by my definition, an outsider. Yes, of course he became an insider later -- that's the dream of of every outsider!

Having an undergrad degree -- or a degree of any kind -- does not make one an insider. What counts is employment: whether one is paid to do the work in question. If you're not (as Einstein wasn't in 1905), you're an outsider.

Comment author: Vaniver 06 July 2012 12:21:14AM 2 points [-]

Whatever the "popular" view, all that matters for my purposes is that Einstein was not employed by a university in 1905 when he developed SR, and thus was, by my definition, an outsider.

Shockley, Bardeen, and Brattain all got PhDs, but invented the transistor while at Bell Labs, i.e. not while employed by universities. I think the "jump through the usual hoops" description shminux is using is a more useful one than the "outsider" description you're using.

Comment author: komponisto 06 July 2012 03:04:22AM 3 points [-]

As I stated in the grandparent, the relevant distinction is whether or not you are paid to do the research, or whether you are forced to do it in your "spare time".

On Bell Labs specifically, see Scott Aaronson:

But what about some of the other spectacular inventions of the last fifty years: the laser, the transistor, the fiber-optic cable, the communications satellite? Didn’t those come from the private sector? As it happens, they came from Bell Labs, which is interesting as the sort of mammoth exception that proves the rule. Because of AT&T’s government-sanctioned monopoly, for much of the 20th century Bell Labs was able to function like the world’s largest university, devoting billions of dollars to “irrelevant” research.

Thus, the people you mention were, I assume, doing their actual jobs when they invented the transistor, which makes them analogous to academics, and not analogous to Einstein in the patent office.

The modern analogue of Einstein would be someone dropping out of grad school, becoming a software developer (or something), and within a few years posting groundbreaking papers on the arXiv that they wrote for the fun of it. You can call such a person an "insider" if you like because of their (unfinished) education, but I guarantee you they sure as hell won't feel like one in the years before their paper comes out. (They won't have library privileges, won't get invited to physicist parties, and generally won't be taken seriously because...they're not a physicist, they're a software developer.)

I think the "jump through the usual hoops" description shminux is using is a more useful one than the "outsider" description you're using.

More useful for what, exactly?

Comment author: Vaniver 06 July 2012 03:33:57AM 2 points [-]

As I stated in the grandparent, the relevant distinction is whether or not you are paid to do the research, or whether you are forced to do it in your "spare time".

I apologize, it appears I didn't read the post in question carefully enough; the criterion of if you're paid to research is a useful one.

But although Einstein is salient, I can't think of too many other examples. The two that leap to mind are Green (working in the early 1800s) and Lavoisier (working in the late 1700s), but from that I would expect "household name scientist who was an outsider" to be something that shows up once or twice a century. (I'm counting Lavoisier because he was funded by his tax farming, but he was definitely part of the 'establishment' of the day.)

(And you do see contemporary outsider contributions if you know where to look, like Gary Cola or Jack Andraka, but not at the household name level- probably because there aren't that many household name scientists!)

Comment author: darius 08 July 2012 03:23:45AM *  1 point [-]

Better examples of outsider-scientists from around then include Oliver Heaviside and Ramanujan. I'm having trouble thinking of anyone recent; the closest to come to mind are some computer scientists who didn't get PhD's until relatively late. (Did Oleg Kiselyov ever get one?)

Comment author: komponisto 08 July 2012 06:08:08AM 0 points [-]

the salient example of Einstein

Better examples of outsider-scientists from around then include Oliver Heaviside and Ramanujan

Again, I don't care whether the person remained an outsider for their entire life; all they need to have done is to have made a contribution while outside. Thus Einstein in the patent office fully counts.

Moreover, it is worth noting that Ramanujan was brought to England by the ultra-established G.H. Hardy, and even Heaviside was ultimately made a Fellow of the Royal Society. So even they became "insiders" eventually, at least in important senses.

Comment author: darius 08 July 2012 10:04:04AM 1 point [-]

In Einstein's first years in the patent office he was working on his PhD thesis, which when completed in 1905 was still one of his first publications. I've read Pais's biography and it left me with the impression that his career up to that point was unusually independent, with some trouble jumping through the hoops of his day, but not extraordinarily so. They didn't have the NSF back then funding all the science grad students.

I agree that all the people we're discussing were brought into the system (the others less so than Einstein) and that Einstein had to overcome negative selection even while some professors thought he showed promise of doing great things. (Becoming an insider then isn't guaranteed -- in the previous century there was Hermann Grassman trying to get out of teaching high school all his life.)

Heaviside and Ramanujan accomplished less than Einstein, but they started way further outside.

Comment author: hairyfigment 05 July 2012 06:14:24AM 2 points [-]

While it filters out some good people, it probably does not reject the very best, otherwise we would see an occasional example of someone making a significant discovery outside academia.

If you still mean physics: why this confidence about the existence of low-hanging fruit? My grad student friend had to go to the LHC to work on (I think) his thesis. I assume they don't let people in off the street.

If you mean academia in general: have you forgotten where you are? ^_^

Comment author: shminux 05 July 2012 03:27:08PM 1 point [-]

If you still mean physics: why this confidence about the existence of low-hanging fruit?

Maybe there is some misunderstanding here. I'm sure there is plenty of low-hanging fruit still undiscovered. But you have to first get to that hard-to-reach orchard where it grows.

My grad student friend had to go to the LHC to work on (I think) his thesis. I assume they don't let people in off the street.

Indeed they don't, though I'm not sure how it is related to my point that negative selection is not a total disaster.

If you mean academia in general: have you forgotten where you are? ^_^

Where am I?

Comment author: Raemon 05 July 2012 04:46:28PM 0 points [-]

On Less Wrong, which has an anti-academia bias.

Comment author: shminux 05 July 2012 06:55:36PM 0 points [-]

Less Wrong, which has an anti-academia bias.

If so, this is rather irrational, given that probably every high-profile/high-status contributor to this forum, with the notable exception of EY, either works in academia or is being/has been trained in academia.

Comment author: wedrifid 05 July 2012 07:33:57PM 5 points [-]

If so, this is rather irrational,

It isn't so. It's more a relative thing---"not quite as extremely biased towards academia as the average group of this level of intellectual orientation can be expected to be".

given that probably every high-profile/high-status contributor to this forum, with the notable exception of EY, either works in academia or is being/has been trained in academia.

Luke has minimal official academic training too. Mind you he is more academic in practice than most people (probably most academics too, come to think of it.)

Comment author: DanArmak 05 July 2012 06:32:03PM 0 points [-]

Why do you call it a bias? Maybe it's being less wrong than others who have a pro-academia bias.

Comment author: hairyfigment 06 July 2012 01:10:57AM 0 points [-]

I'm not sure how it is related to my point that negative selection is not a total disaster.

What would look different if it were? (Aside from, say, the reduced chance of someone finding the Higgs.)

Comment author: shminux 06 July 2012 03:09:30AM 0 points [-]

Then I would expect that once in a while some filtered out genius discovers something really exciting, against all odds, as I mentioned already.

Comment author: pnrjulius 06 July 2012 10:57:35PM 0 points [-]

otherwise we would see an occasional example of someone making a significant discovery outside academia.

Should we all place bets now that it will be Eliezer?

Comment author: shminux 07 July 2012 12:23:52AM 3 points [-]

He repeatedly mentioned that his skill is in formulating theorems, not proving them, and he has not formulated even one after some 5 years of working on the same problem, so the chances are not good.

Comment author: pnrjulius 06 July 2012 10:44:02PM *  3 points [-]

I don't think it's quite true that "fail once, fail forever", but the general point is valid that our selection process is too much about weeding-out rather than choosing the best. Also, academic doesn't seem to be very good at the negative selection that would make sense, e.g. excluding people who are likely to commit fraud or who have fundamentally anti-scientific values. (Otherwise, how can you explain how Duane Gish made it through Berkeley?)

Comment author: Eliezer_Yudkowsky 05 July 2012 02:13:26AM 4 points [-]

What my externally observable percentiles look like:

  • Writing: 99+%
  • Math: 99+%
  • Conceptual originality: 99+%
  • Programming: 95%
  • Conformity / ability to obey incorrect orders: 20%

What my educational credentials look like:

  • Highest level of education completed: 8th grade
Comment author: MBlume 05 July 2012 05:48:14AM 12 points [-]

Programming: 95%

I'm with J and Alex -- are you comparing yourself to people or to programmers? I'm fairly sure FizzBuzz puts one well above 95% in the general population

Comment author: Eliezer_Yudkowsky 05 July 2012 07:04:09AM 0 points [-]

Programmmers.

Comment author: cousin_it 05 July 2012 12:47:29PM *  17 points [-]

Is the 99+% for math also compared to mathematicians?

Comment author: Eliezer_Yudkowsky 05 July 2012 06:04:35PM 6 points [-]

That'd be a much harder question to answer; my talent is specialized toward figuring out the shape of the right theorem to be proved, not the actual proof which is where most modern math concentrates its prestige. (This is an objectively verifiable form of mathematical talent; it means that sometimes Marcello would prove something and I would look at it and say, "That doesn't look right" and at least half the time there'd be a mistake.) I feel insecure about not being an expert in the tools by which most modern mathematicians measure basic competence; I can also apparently make "well, if that's your problem, try transforming it this way" suggestions to someone doing graduate mathematical research at Yale that are accepted as brilliant. I confidently depose that, even taking unusual talents into account, I am not in the literal top tier of mathematical potential - if I can explain basic Bayes better than anyone or was first to state the Lob problem or invented TDT, those outputs drew on at least some non-mathematical high-percentile sections of my brain (explanatory ability in the first case, or what's somewhat vaguely referred to as "philosophical" talent in the other two). I'm also reasonably confident that, given a hundred modern mathematicians, an average of zero will pick the right problem to solve.

I think I'm comfortable at this point with saying that I'm in the top 99+% of writers - I've been picking up "real" books and trying to read them and finding that they seem visibly badly-written to me now that I've written HPMOR. Though I'm still not in the literal top tier; there are basic things in writing that I still don't do too well, despite being outstanding in others, and my new level of skill is just enough to start noticing that Neil Gaiman and Terry Pratchett are doing things way the hell above me.

Comment author: DanArmak 05 July 2012 06:47:16PM 11 points [-]

That'd be a much harder question to answer

"What did you mean when you said 99%" should not be a hard question to answer. "Which alternative is correct" may be hard, but did you have in mind "one alternative is correct but I don't know which"?

Comment author: NancyLebovitz 07 July 2012 01:05:52PM 4 points [-]

Have you written anything about the process of figuring out the shape of the theorem to be proved, and/or identifying the plausibility of a theorem?

Comment author: Tyrrell_McAllister 05 July 2012 09:46:48PM *  2 points [-]

That'd be a much harder question to answer; my talent is specialized toward figuring out the shape of the right theorem to be proved, not the actual proof which is where most modern math concentrates its prestige.

Being able to give some actual proofs is a prerequisite of prestige. But it's not clear to me that it's right to say that mathematics concentrates its prestige there. See, for example, Fields Medalist Timothy Gower's article The Two Cultures of Mathematics (pdf):

The “two cultures” I wish to discuss will be familiar to all professional mathematicians. Loosely speaking, I mean the distinction between mathematicians who regard their central aim as being to solve problems, and those who are more concerned with building and understanding theories.

...

Let me now briefly mention an asymmetry similar to the one pointed out so forcefully by C. P. Snow. It is that the subjects that appeal to theory-builders are, at the moment, much more fashionable than the ones that appeal to problem-solvers.

Comment author: komponisto 05 July 2012 09:59:04PM 0 points [-]

I suspect the distinction Eliezer is making is more akin to the controversial "theoretical vs. experimental" one proposed by Jaffe and Quinn than the traditional "theory-builder vs. problem-solver" one discussed by Gowers.

Comment author: Tyrrell_McAllister 05 July 2012 10:31:39PM 0 points [-]

It's been years since I read the Jaffe–Quinn article. But, as I recall, it was more about the methods used to answer questions, and about how rigorous human-verifiable proofs might give way to heuristic/probabilistic and computer-aided proofs. Eliezer, on the other hand, seemed to be saying that mathematicians concentrate prestige on answering questions (by whatever means the community considers to be adequate), as opposed to "figuring out the shape of the right theorem to be proved".

Comment author: komponisto 05 July 2012 10:45:36PM 0 points [-]

Jaffe and Quinn mainly advocate that labor should be divided between people who make conjectures ("theoreticians") and people who prove them ("experimentalists"). I don't think there is much of anything about probabilistic or computer-aided proofs.

Comment author: Tyrrell_McAllister 05 July 2012 11:31:01PM *  1 point [-]

You are right. Looking at the Jaffe–Quinn paper again, it is closer to the distinction that Eliezer was making. (However, I note that the mathematical "theoreticians" in that article are generally high-prestige, and the "rigorous mathematicians" have to fight the perception that they are just filling in details to results already announced.)

My mischaracterization of Jaffe and Quinn's thesis happened because (1) Thurston replied to their article, and he discusses computer-aided proofs in his reply; and (2) even more embarrassingly, I conflated the Jaffe–Quinn article with the Scientific American article The Death of Proof, by John Horgan.

Comment author: Sewing-Machine 06 July 2012 09:28:12PM *  1 point [-]

figuring out the shape of the right theorem to be proved, not the actual proof which is where most modern math concentrates its prestige.

Many geniuses have a bad reputation for checking all the details and writing them down. Kontsevich and Thurston are good examples. People complain about this but the status hierarchy isn't much affected by those complaints. Low-status mathematicians don't often get away with Thurston's attitude, but nor do they accumulate status by being more conscientious.

Comment author: CronoDAS 06 July 2012 05:13:22PM 1 point [-]

my new level of skill is just enough to start noticing that Neil Gaiman and Terry Pratchett are doing things way the hell above me.

I once got this feeling reading Stephen R. Donaldson's The Runes of the Earth - that this was a level of writing that was way beyond what I could see myself reaching. Oddly, I didn't get this feeling when reading Terry Pratchett, even though I still think that Terry Pratchett is probably a better writer than, say, Shakespeare.

And I don't know what people see in American Gods - I've read over one hundred books I think were better. And I mean that literally; if I spent a day doing it, I could actually go through my bookshelves and write down a list of one hundred and one books I liked more. I couldn't do that for most of Terry Pratchett's novels.

Comment author: Eliezer_Yudkowsky 06 July 2012 08:06:04PM 1 point [-]

I also tried American Gods for a while and found that its charm was mostly lost on me - maybe I didn't get far enough in. Good Omens, on the other hand...

Understand, I always knew that Good Omens was a great book and that I wasn't yet writing that well; it's only now that I'm staring at a Neil Gaiman short story, thinking, "I can tell that he's doing something outstandingly right here that I'm not doing, but it's hard to say exactly what..."

Comment author: NancyLebovitz 07 July 2012 01:02:50PM 1 point [-]

Gaiman frequently doesn't grab me, though I think "A Study in Emerald" is brilliant.

I wish American Gods had been written by someone who understood and liked America better. Why was the computer god a marketing monster rather than a programmer? Or a computer? And I know it's not fair to blame a writer for not writing a different book, but I'd like to see a version of the idea with the guts to engage with actual American religions.

Comment author: TheOtherDave 06 July 2012 08:58:12PM 1 point [-]

American Gods is pretty evenly written; if it didn't grab you in the first fifty pages or so it was probably never going to. (I say this as someone who fell in love with it and considers it among my favorite of his work.)

Comment author: CronoDAS 07 July 2012 12:14:05AM 1 point [-]

I blame the SeinfeldIsUnfunny effect. I've seen GodsNeedPrayerBadly done so many other times that it seemed like a cliche.

Comment author: Desrtopa 07 July 2012 03:02:36AM 1 point [-]

Personally, I disliked that trope even before I'd seen it enough times for it to seem cliche, but I count American Gods among my favorites of Gaiman's work in spite of it.

Comment author: TheOtherDave 07 July 2012 01:12:47AM 0 points [-]

Perhaps. It was far from being my introduction to that trope, but I found it worth reading for something other than the originality of that particular idea. Still, different people like different things in their art.

Comment author: Alicorn 06 July 2012 05:34:50PM 1 point [-]

Oddly, I don't like Gaiman much at all on his own, and I don't like large doses of Pratchett either, but I loved Good Omens - they balanced each other's weaknesses.

Comment author: MBlume 06 July 2012 08:01:13PM 3 points [-]

they balanced each other's weaknesses.

Not at all like this

Comment author: [deleted] 09 July 2012 07:45:28AM 0 points [-]

Oh wow, I have the exact opposite reaction; I love both Gaiman and Pratchett separately, but dislike Good Omens-- they undercut each other's strengths.

I tend to describe it as: "They have a similar worldview, but Gaiman is dark, whereas Pratchett is light. When you put them together the result is a rather bland gray."

Comment author: randallsquared 16 July 2012 02:13:09PM 0 points [-]

I've read over one hundred books I think were better. And I mean that literally; if I spent a day doing it, I could actually go through my bookshelves and write down a list of one hundred and one books I liked more.

I've read many, many books I liked more than many books which I would consider "better" in a general sense. From the context of the discussion, I'd think "were better" was the meaning you meant. Alternatively, maybe you don't experience such a discrepancy between what you like and what you believe is "good writing"?

Comment author: CronoDAS 16 July 2012 10:46:16PM 1 point [-]

A book can be well written and still be bad because of other flaws. Nathaniel Hawthorne's The Scarlet Letter was very well written in a technical sense, but the story itself was boring as hell and Hawthorne's skill couldn't save it.

Comment author: Desrtopa 06 July 2012 08:46:06PM 0 points [-]

I think I'm comfortable at this point with saying that I'm in the top 99+% of writers - I've been picking up "real" books and trying to read them and finding that they seem visibly badly-written to me now that I've written HPMOR. Though I'm still not in the literal top tier; there are basic things in writing that I still don't do too well, despite being outstanding in others, and my new level of skill is just enough to start noticing that Neil Gaiman and Terry Pratchett are doing things way the hell above me.

What of Pratchett's work are you judging by? Middle-Pratchett is top tier, but I'm much less impressed with recent-Pratchett (going back at least a few years before the Alzheimer's diagnosis.)

I get the feeling that a lot of the top experts at writing aren't necessarily experts on writing, because they can and do lose their touch without noticing that something is wrong.

Comment author: DanArmak 05 July 2012 06:43:41PM 4 points [-]

What published software have you written that I should look at? A quick Google search comes up with this, listing Flare and Document Designer. Is there something different you'd rather be known for and judged by?

Incidentally, Googling "Eliezer Yudkowsky software" also returns this link, which is someone's search on pricegrabber.com for "Rationally Speaking Eliezer Yudkowsky Bayles" [sic] for which pricegrabber returns the Dragon Naturally Speaking software. I don't know why this is near the top of Google results, but it Rationally Speaking sounds like a real-life Eliezer Yudkowsky Fact...

Comment author: wedrifid 05 July 2012 07:47:02PM 1 point [-]

What published software have you written that I should look at? A quick Google search comes up with this, listing Flare and Document Designer. Is there something different you'd rather be known for and judged by?

95% of programmers don't have published personally published software.

Comment author: fubarobfusco 05 July 2012 11:56:08PM 3 points [-]

A more general form of the question: "What publicly available evidence is there for this 95th percentile claim?"

Comment author: thomblake 05 July 2012 08:09:43PM 1 point [-]

95% of programmers don't have published personally published software.

Really?

Comment author: gwern 05 July 2012 11:51:41PM 4 points [-]

The Wikipedia article on GitHub claims >1m users (passing the mark in 2011), with around 90,000 unique repositories out of 2m. StackOverflow has a few relevant questions like http://stackoverflow.com/questions/453880/how-many-developers-are-there-in-the-world which give me a vague estimate of perhaps 5-15 million worldwide. 0.1m unique repos to 15m developers is 6.6% and roughly consistent with 5%.

On the other hand, I don't know what 'personally published software' might be. A complete standalone executable or library? I could see most programmers confining their efforts to working on existing codebases, yes, and this would likely cut down the GitHub repo estimate a lot since many of them are no doubt forks with a patch or two of some main repo or dead ends of various kinds.

Comment author: ShardPhoenix 06 July 2012 04:15:03AM *  2 points [-]

You're off by an order of magnitude - it's 0.66%.

Comment author: gwern 06 July 2012 04:22:53AM 0 points [-]

Whups. I just thought 1/15... Well, an order is still within the margins of error here since a GitHub repo count ignores other sites like SourceForge.

Comment author: wedrifid 05 July 2012 08:21:41PM 2 points [-]

Really?

Yes, I'm completely certain. (ie. p=0.6)

Comment author: DanArmak 05 July 2012 07:51:49PM 1 point [-]

Are you implying he's better than 95% just because he has published something? Or what do you mean?

Comment author: wedrifid 05 July 2012 07:53:48PM *  0 points [-]

Are you implying he's better than 95% just because he has published something?

No, I'm saying that it would be misleading to imply the reverse if someone hadn't or to place all that much weight on the published software if they have (except in as much as the published personal software projects establish a lower bound.)

Comment author: DanArmak 05 July 2012 09:00:11PM *  0 points [-]

Well, that's why I asked him if he thought he could be fairly judged based on those two published projects. I didn't go ahead and just judge them, and I won't unless Eliezer says they're worth judging because they do establish such a lower bound for him.

Comment author: J_Taylor 05 July 2012 03:27:05AM 9 points [-]

What population are you comparing yourself to?

Comment author: Ezekiel 05 July 2012 06:26:47PM 9 points [-]

Are these your own estimates, or have you found some objective, accurate test for ranking "Conceptual originality"?

Comment author: Eliezer_Yudkowsky 06 July 2012 01:30:57AM 4 points [-]

I put that in because I didn't think any non-trolls would seriously dispute the 99+% part, not because I knew how to measure it down to the sixth decimal place.

Comment author: CronoDAS 06 July 2012 01:48:45AM 2 points [-]

Highest level of education completed: 8th grade

These days, you can claim to be homeschooled. ;)

Comment author: NiklasF 24 July 2012 10:31:59AM -3 points [-]

This is a too bold claim. So your writing skill is at the nobel prize class? Your math is close to fields prize level? And your programming skills is better than programmers who has spend a life-time programming?

You're 32 and aldready a better mathematician, programmer and writer than people spending life times doing that? There's only 0.1-% people in each category that are better than you?

Instead of saying you're among the worlds best, bragging ego inflation. Wouldn't have sufficed to say, "Im good at a,b,c... even tough i've only had 8th grade education"?

Comment author: MarkusRamikin 24 July 2012 10:40:53AM 2 points [-]

So your writing skill is at the nobel prize class?

I don't think that's what 99+% means.

Comment author: AlexMennen 05 July 2012 04:30:22AM 0 points [-]

Programming: 95%

Why so low?

Comment author: Dreaded_Anomaly 19 July 2012 08:21:20PM 1 point [-]

I just now read an interview which brings up the rise of negative selection in job applications:

In the past, they wanted lots of applicants, so now they’re overwhelmed by applicants, so now every company will tell you they’re getting thousands or tens of thousands of applicants for positions. You couldn’t possibly screen them all by hand, because you can’t look at them all, so they use automated systems to do the screening. But the screening is never as good as somebody who has human judgment, and the way screening works is you build in a series of typically yes/no questions that try to get at whether somebody has the ability to do this job. And a lot of that ultimately ends up, it’s all you can ask about, is experience and credentials. So you end up with a series of yes/no questions. And you have to clear them all, and I think people building these don’t quite understand that once you have a series of these yes/no questions built in, and the probabilities are cumulative right? You have to hit them all, then you pretty easily end with no one that can fit.

Comment author: thomblake 19 July 2012 08:26:21PM 2 points [-]

You have to hit them all, then you pretty easily end with no one that can fit.

Worse, for the ones that do, you're probably just responding to noise. If it's very improbable that any applicant will really match all of the screening criteria, then that can become smaller than the probability of a false positive.

Comment author: Oscar_Cunningham 24 July 2012 12:52:38PM *  0 points [-]

But the screening is never as good as somebody who has human judgment

That's weird, normally human judgement is worse than simple measures.

Comment author: CronoDAS 06 July 2012 08:55:49AM *  1 point [-]

It’s apparently so important that people really care about performance – as opposed to, say, in medicine, where we exclude brilliant doctors if they don’t have the stamina to work ninety hours a week.

How much does this actually matter, I wonder? Is there really that big a difference between the best doctor in a group of 100 and the 10th best doctor in that same group? (The 10th best golfer in a tournament doesn't take home the trophy, but the 10th best doctor in the hospital can still do a fine job treating a broken arm.)

Comment author: eg220 11 July 2012 10:02:30PM 0 points [-]

In some examples, like elite college admission, it seems more like there are both negative and positive controls. Although colleges use SAT scores and GPAs to weed out people who aren't "good enough," they also look at whether students are exceptional through supplemental essays or awards the student received in high school. Negative controls bar many students from admission, but in many cases, positive controls must also be used to select a final class out of acceptable applicants. I'm not in academia, but it seems similar. Although you may get off the "track," you still need to do quality research to get a good position at the end of it. Even in top graduate programs, not all PhDs from those programs will end up finding academic jobs. At some point, you have to do more than just not fail.

Comment author: [deleted] 07 July 2012 02:39:11PM *  0 points [-]

Academia is another example of negative selection.

This doesn't seem like a fair generalization. At the undergraduate level, selection procedures are (and are rightly) negative. At the graduate level, things are very different across different fields and departments, but on the whole I think graduate admissions is a mix: people are weeded out until you have a small group of acceptable candidates, and then the exceptional ones are pulled out on the basis of their specific work. At the tenure level, you get a similar mix but there it's very heavily in the direction of positive selection. Almost all of what a tenure committee is thinking about is the quality of your work (there are unfortunate exceptions to this, no doubt). Journal and book publishing are to an even greater extent a positive selection system, though again there are some unfortunate exceptions (like law).

So on the contrary, I think academia is largely a positive selection system. You're description of the steps of academic success is accurate, but doesn't provide evidence in the direction of positive or negative selection systems.

And one should also remember that positive selection systems tend to be enormously costly. With something like undergraduate admissions to sizable universities, positive selection is practically impossibile. Graduate admissions, where positive selection is at least present in the process (though it is still largely negative) requires months of work from a substantial portion of the whole department. Tenure hearings are also months in the making, but there you're evaluating a single person.

So don't knock negative selection too hard.

Comment author: johnlawrenceaspden 06 July 2012 12:02:04PM 0 points [-]

Very thought-provoking. Thank you!

How would we tell the difference between 'positive' and 'negative' selection? If I imagine that I've got two scores x and y (0<x,y<1) which I've got from various sources and consider equally important, is something like 'accept if x>0.9 AND y>0.9' what you mean by 'negative selection', and 'accept if x>0.995 OR y>0.995' positive selection?

If I'm thinking along the right lines here, is there a general principle (like 'acceptance set must be convex'?), or do I have the wrong end of some crucial stick?

Comment author: johnlawrenceaspden 06 July 2012 12:23:40PM 0 points [-]

To be a bit less abstract, if x and y are sportiness and intelligence, and they're uniform, then the AND rule gives you bright, fit people (i.e. the sort you naturally like), with a few superstars, whereas the OR rule gives you uber-nerds who may or may not be sporty and uber-jocks who may or may not be bright.

In fact it also might depend on how x and y are distributed. If they're uniform then the difference between the AND and the OR rule feels less pronounced than if they're gaussian. I think that the gaussian/AND case is going to give you very few people who are good at both, whereas the uniform scores case gives you some.

Comment author: brilee 06 July 2012 03:23:15AM 0 points [-]

One word:

China.

One massive examination that determines your entire future? Isn't that about as positive selection as you can get?

Comment author: AlexMennen 06 July 2012 03:50:58AM *  3 points [-]

No. Positive selection means taking anyone who performs exceptionally well on any one out of many possible measures, whereas negative selection means taking anyone who performs at least passably on all of many measures. On the scale of how to integrate the information you get from each test, there only is one test, so you can't distinguish between positive and negative selection.

On the scale of what the test itself measures, most exams tend to provide negative selection because the students usually know most of the material, so the smart kids can reliably get more than 90% of the questions correct, and the best way to do well involves having as few weak points as possible, rather than having some exceptionally strong abilities. So unless the one massive examination that they take in China is the Putnam, it's probably more like negative selection.

Comment author: jsteinhardt 08 July 2012 07:58:33AM 0 points [-]

Well, it is pretty hard from what I hear. I predict the hardest problems on the exam are as hard as easy-medium Putnam problems.

Comment author: CronoDAS 06 July 2012 02:56:48AM 0 points [-]

Elite college admissions is an example of a negative selection test. There’s no one way you can do really, really well, and thereby be admitted to Harvard.

I suspect that "being rich enough to make a sufficiently large donation" will get you in (as long as you've got a high school diploma or GED). "Sufficiently large" may run in the hundreds of millions of dollars, though.

Comment author: Vaniver 06 July 2012 03:44:44AM 0 points [-]

Rumor has it that it takes $5M to get accepted off the waitlist. If you don't get waitlisted... I'm sure that something can be arranged, but think your expectation is probably a reasonable estimate.

Comment author: CronoDAS 06 July 2012 04:20:38AM *  1 point [-]

I was thinking of the scene in the Rodney Dangerfield movie "Back to School" in which Rodney Dangerfield's character, who became a wealthy businessman despite lacking a high school diploma, tries to enroll in the college in which his son is about to drop out of.

Wikipedia's description of the scene in question:

Possessing neither a high school diploma nor any transcripts or SAT scores, Thornton’s efforts seem to be stalled. But when the university's "Dean" Martin -– a play on Dean Martin -– asks how he can possibly admit an unqualified student, the next scene cuts to a groundbreaking of the university's new Thornton Melon School of Business.