LESSWRONG
LW

Comment Permalink
Kaj_Sotala140

Intelligence seems relatively static, but AFAIK once you've reached a certain minimum threshold in intelligence, conscientiousness becomes a more important factor for actual accomplishment. (Anecdotally and intuitively, conscientiousness seems more amenable to change, but I don't know if the psychological evidence supports that.)

Reply
Barry_Cotter60

Wait, there's real evidence of durable changes in conscientiousness? Point me its way. The psychology literature does not appear (after a brief search) to support the idea of lasting change. I would be happy to be wrong.

Reply
See in context
by lukeprog
1 min read
75

10

Series: How to Purchase AI Risk Reduction

Here is yet another way to purchase AI risk reduction...

Much of the work needed for Friendly AI and improved algorithmic decision theories requires researchers to invent new math. That's why the Singularity Institute's recruiting efforts have been aimed a talent in math and computer science. Specifically, we're looking for young talent in math and compsci, because young talent is (1) more open to considering radical ideas like AI risk, (2) not yet entrenched in careers and status games, and (3) better at inventing new math (due to cognitive decline with age).

So how can the Singularity Institute reach out to young math/compsci talent? Perhaps surprisingly, Harry Potter and the Methods of Rationality is one of the best tools we have for this. It is read by a surprisingly large proportion of people in math and CS departments. Here are some other projects we have in the works:

  • Run SPARC, a summer program on rationality for high school students with exceptional math ability. Cost: roughly $30,000. (There won't be classes on x-risk at SPARC, but it will attract young talent toward efficient altruism in general.)
  • Print copies of the first few chapters of HPMoR cheaply in Taiwan, ship them here, distribute them to leading math and compsci departments. Cost estimate in progress.
  • Send copies of Global Catastrophic Risks to lists of bright young students. Cost estimate in progress.

Here are some things we could be doing if we had sufficient funding:

  • Sponsor and be present at events where young math/compsci talent gathers, e.g. TopCoder High School and the International Math Olympiad. Cost estimate in progress.
  • Cultivate a network of x-risk reducers with high mathematical ability, build a database of conversations for them to have with strategically important young math/compsci talent, schedule those conversations and develop a pipeline so that interested prospects have a "next person" to talk to. Cost estimate in progress.
  • Write Open Problems in Friendly AI, send it to interested parties so that even those who don't think AI risk is important will at least see "Ooh, look at these sexy, interesting problems I could work on!"

 

 

Personal Blog

10

Mentioned in
Reaching young math/compsci talent
New Comment
75 comments, sorted by
top scoring
Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings
[-]JoshuaZ250

Send copies of Global Catastrophic Risks to lists of bright young students

This may come across as spamming and will likely send crank signals.

Reply
[-]Nisan160

I dunno. It's a book. If anyone sends me a book, I'll consider keeping it and likely look at the first couple of pages, even if it's Dianetics or The Book of Mormon. I don't regard books as the physical and memetic pollution that spam is.

If I got a vanity-press book like this, I'd regard it as cranky non-spam. But Global Catastrophic Risks is published by Oxford University Press, which matches the pattern "legitimate" rather than "crank".

If I got a religious text, I'd be unimpressed. But Global Catastrophic Risks differs from a religious text in that it's a collection of essays by different authors who no doubt disagree about many things, rather than a canonized text that's regarded as perfect. And the hidden agenda of someone who gives me Global Catastrophic Risks is to get me thinking about global catastrophic risks — which is pretty reasonable, although not universally compelling. It would be much less creepy than receiving a Bible.

In summary, while JoshuaZ might have been turned off by receiving a book when he was a mathematically talented youth, I wouldn't have. So, that's two data points.

tl;dr: Please send me a copy of the Book of Mormon.

Reply
2Alicorn
Do you actually want a copy of the Book of Mormon? It's online, and I bet you could get a free one by filling out a form on an LDS website.
2Nisan
You're right, I can request a free Book of Mormon delivered to my doorstep by missionaries. I notice some mediocre reasons not to do this, as well as some strong unreasonable aversions to doing this; which means maybe I should do it...
7Alicorn
The (plenty of) Mormons I have met are all really nice, friendly people. Missionaries on duty might be different (more goal-oriented, probably) but I bet you they're still nice. Unless you're just concerned about wasting their time, what are your strong unreasonable aversions?
1Nisan
It feels dishonest because if their goal is to convert me, calling on me is a waste of time. I know I'm not going to convert. I can satisfy this aversion by being honest up-front about not being likely to convert. I bet if I tell them I nevertheless want to keep the book and look at it a little, they'll still come over and give it to me. That way I'm being honest; and from a more consequentialist point of view I'm not wasting anyone's time but my own because converting people to Mormonism isn't a great goal anyways. Another aversion is telling me that I'm going to disagree strongly with what they have to say, and that suppressing disagreement will be awkward and that expressing disagreement will be rude. I can respond to this aversion by deciding beforehand how to approach the conversation: I could either have the goal of learning what Mormonism means to them, or I could practice expressing disagreement. An actual reason for not casting Summon Mormon Missionaries is that the LDS Church will have my contact info forever, and will pester me in the future. If I can remove that cost, I'll do it.
9MileyCyrus
Speaking from experience, they won't. I called them once for a free Book of Mormon. They came over and I said thanks for the book and but I don't want to convert. They made a follow-up call, but I haven't heard from them since.
2Alicorn
Hee hee hee. The contact info thing probably is an actual problem and a legit reason to hold back.
2arundelo
If you really want a paper Book of Mormon but want to avoid interacting with people who will try to convert you I recommend checking a store that sells stuff that was donated to it (e.g., Goodwill or Salvation Army in the US). These places often sell books for a dollar or less. (Because Mormons like to give copies of the Book of Mormon away, there are a lot of them that get taken to thrift shops when non-Mormons get rid of old stuff; because of supply and demand, I think you're less likely to find a copy in a regular used bookstore.) (Insults to people's holy books rot13ed out of a possibly excessive sense of politeness: Or jnearq; abg sbe abguvat qbrf vg vapyhqr n Obbx bs *Rgure*, nf sebz jung yvggyr V'ir ernq bs vg, vg vf *rira zber obevat* guna gur Wrjvfu naq Puevfgvna fpevcgherf.)
0wedrifid
Their goal is to earn their heavenly reward (or possibly 'new earth' reward, not sure on the details). The heavenly reward scheme is not based on commission but on the work that they do so you are not doing them a disservice.
0Nisan
The test for dishonesty I'd use here is: Would a missionary (or their superiors in the Church) be dismayed if they learned that a potential new latter-day saint had been leading them on? I suppose the answer is yes, no matter the theology.
1Paul Crowley
I predict that if, when you ask for the book, you say "there is zero chance of me converting, I just want it for reference", they will send it to you, and follow up, anyway.
0Jayson_Virissimo
I tried getting a free copy of the Koran from here, but it never arrived. IDK why.
[-]komponisto180

This is basically my approach of choice, and I am very happy to see SI pursuing it. That said, I would like to make a couple of comments:

Specifically, we're looking for young talent in math and compsci, because young talent is...(3) better at inventing new math (due to cognitive decline with age).

So, if Edward Witten (age 60)* called you up tomorrow and said he was interested in working on Friendly AI, you would tell him to get lost? I think not. At least, I hope not.

I'm not saying you should target older people in your recruitment activities. (As if that were even possible.) But I am strongly advising against getting into any kind of mindset where you would end up closing the door on any mathematically accomplished people who happen to see the light on this matter.

AGI really might be decades or more away. The people who are "young" now won't be that way forever. You may want their help in the future. In particular, you may want the help of a future John Baez, who after a satisfying run in more mainstream topics, decides at age 40 to turn their attention to "helping humanity" -- only in the form of FAI research rather than environmentalism.

(Also, if you b... (read more)

Reply
8lukeprog
No, obviously not. We're targeting young people, but that doesn't mean we're closed to older people.
7Fhyve
I am a young person (20) who is good at math and hasn't been entrenched in the system yet. I am also already on board with AI risk reduction. I would really like to work as a researcher. However, I don't have much to show for myself, and I don't think I can substantiate my claims right now. I do not know enough about research to know if I am going to be good at it. At the moment, I have a pretty good topical view of math, but not a very good technical view - I am only into second year university math. Pure math and theoretical comp sci especially appeal to me. How do I find out if I can be a researcher? How do I show you that I can be a good researcher if I find that I can in fact become a good researcher? What sort of math should I be studying - any textbooks to recommend?
3lukeprog
Thanks for your interest! Please contact louie.helm [at] singinst.org.
2incariol
You can find a few suggestions here, for starters.
0DaFranker
I was reading this and preparing to post a questions-comment just like his, so thanks!
3Risto_Saarelma
I'd like to see more counterarguments to the thing about mathematicians being much less useful for ground-breaking work after their 20s that don't rely on extreme outliers like Witten, Andrew Wiles or Paul Erdös.
[-]komponisto100

That would be difficult, since "groundbreaking work" automatically implies "extreme outlier".

In fact, I would expect that typical mathematicians are much more useful above 30 than below -- to a greater extent than is the case for the extreme outliers.

Reply
[-]Kaj_Sotala220

Simonton (1988) Age and Outstanding Achievement: What Do We Know After a Century of Research? Psychological Bulletin, Vol. 104, No. 2, 251-267.

Short version: the productivity for mathematicians seems to peak around late 20s or early 30s, with the productivity after the peak falling to less than one-quarter the maximum. However, the average quality of a contribution does not seem to vary with age, and exceptional researchers (in any field) tend to remain unusually profilic, as compared to an average researcher of the same age, even after passing their peaks.

Long version:

In the first place, the location of the peak, as well as the magnitude of the postpeak decline, tends to vary depending on the domain of creative achievement. At one extreme, some fields are characterized by relatively early peaks, usually around the early 30s or even late 20s in chronological units, with somewhat steep descents thereafter, so that the output rate becomes less than one-quarter the maximum. This agewise pattern apparently holds for such endeavors as lyric poetry, pure mathematics, and theoretical physics, for example (Adams, 1946; Dennis, 1966; Lehman, 1953a; Moulin, 1955; Roe, 1972b; Simonton, 1975

... (read more)
Reply
[-]Kaj_Sotala160

...and after posting that comment, I remembered that I had made an earlier post citing studies that said that it's the middle-aged and not young scientists who are the most productive, which is in conflict with the results I just quoted. I feel silly now. I guess I should re-read the studies that I referenced three years ago to figure out what version is correct.

Reply
8gwern
Just to make the obvious point, your earlier post seems to draw on citations using mostly post-60s and later data, while that 1988 paper uses many citations from the 60s or earlier.
8John_Maxwell
If old and young mathematicians have different strengths and weaknesses maybe it's best to have a few of both.
6Kaj_Sotala
(part 2)
1Risto_Saarelma
Depends on the surface area of unbroken ground. I understand there are quite a few marginal areas in mathematics where you can come up with novel approaches that will be quite impressive to the other five people working on that specific sub-sub-sub-area, but not necessarily that much to mathematics at large. Also, contemporary mathematicians whose names are actually recognizable by popular science literate non-mathematicians are a very small group even compared to the sort of top researchers who are working with the sort stuff the apocryphal wisdom about needing to be in your 20s seems to apply to. Though I'd also like to see more arguments about how above 30 mathematicians can do all sorts of useful stuff when you don't get fixated on paradigm-upending world-class results, and what sort of stuff this is.
0Vaniver
Galenson's book on artists fascinated me: he identified two clusters, experimental artists who liked to sketch and rework things and whose quality increased with age, and conceptual artists, who liked doing preparatory work and outsourcing the actual production, who made massive contributions when young but whose productivity rapidly tapered off. With art, there's room for both types, but I imagine that math and related fields are heavily biased towards the conceptual style, especially the theoretical components of those fields.
6jsteinhardt
Actually, one of the first things that new researchers have to learn is that just thinking about a problem and coming up with ideas will get you nowhere -- you have to actually get your hands dirty and try things out to make progress.
0Vaniver
Oh, definitely. I don't mean to imply that, say, Warhol never got his hands dirty- but that Rembrandt's skill was in the realm of dirty hands and that Warhol's skill was in the realm of insight. (I know in my research the act of sitting down and writing out an idea or sitting down and coding an algorithm or sitting down and going through the math has been indispensable, and strongly recommend it to anyone else.)
[-]Zetetic110

Anything for undergrads? It might be feasible to do a camp at the undergraduate level. Long term, doing an REU style program might be worth considering. NSF grants are available to non-profits and it may be worth at least looking into how SIAI might get a program funded. This would likely require some research, someone who is knowledgeable about grant writing and possibly some academic contacts. Other than that I'm not sure.

In addition, it might be beneficial to identify skill sets that are likely to be useful for SI research for the benefit of those who might be interested. What skills/specialized knowledge could SI use more of?

Reply
4lukeprog
SPARC for undergrads is in planning, if we can raise the funding. See here.
1Zetetic
Awesome, glad to hear it! Alright, I think I'll sign up for that.
[-]JoshuaZ20

Run SPARC, a summer program on rationality for high school students with exceptional math ability. Cost: roughly $30,000.

Do we have any reason to believe that such a program will be more effective than existing summer programs like Ross and PROMYS?

Reply
5lukeprog
Compare their syllabi. Ross and PROMYS don't teach what SPARC is teaching, and they don't put these young students into contact with us. As for effectiveness... what measure of effectiveness did you have in mind?
[-]VincentYu280

SPARC and a number of mostly homogeneous math camps are all looking for pre-college students with strong mathematical ability. Since SPARC's syllabus is notably different from that of math camps, it seems like a bad idea to compete with these camps for the top students. But competition is inevitable if SPARC runs at the same time as these camps; below I have found and listed the 2012 start and end dates for the most prominent math camps:

  • Ross: June 18 – July 27
  • PROMYS: July 1 – August 11
  • HSMC: June 17 – July 28
  • Mathcamp: July 1 – August 5
  • HCSSiM: July 1 – August 11
  • SUMaC: July 15 – August 11
  • RSI*: June 24 – August 4
  • (SPARC: August 6 – August 13)

SPARC's starting date this year conflicts with the end dates of three of these seven camps. Perhaps there are other scheduling constraints, but if not, wouldn't it be a good idea to run SPARC a week later to avoid conflicts? (It is too late to change this year, of course.)

*I know RSI is not a math camp in the spirit of the others, but it's well-known and attracts some students away from math camps.

ETA: And since SPARC is free and relevant to math students, if it can guarantee that it will not conflict with the other program dates, I think... (read more)

Reply
6JoshuaZ
I'm not sure what would be a good metric. But it isn't obvious to me that having a separate program like this is at all likely to be better than having kids go through other programs that teach a lot of math systematically and then snagging some of the kids up when they get a little older. This is especially the case because the existing programs have very good teaching, lot of long-term institutional knowledge, and much better funding. To really effectively run a summer program that attracts top talent you'll need a lot more money. Incidentally, the website could use some work. Obvious things that kids and parents are thinking about when they look at a summer program, costs, dorming, how to apply, dates of the program, should be direct rather than vaguely answered on an FAQ.
5lukeprog
We are reworking the website, we just needed to get something up quickly. Also, we already maxed out our capacity for top young talent by mailing written invitations directly to a bunch of the people we wanted to apply.
[-]John_Maxwell110

You guys should have a simple mailing list to sign up for to get reminded about future camps, and maybe even to broadcast camp related materials (e.g. "here are video lectures from the camp you missed").

Reply
1lukeprog
Yes that will be part of the new CFAR website we're working on.
4paulfchristiano
Cost, logistics, and how to apply were all discussed on the front page, until the application process closed and they were replaced with "the application process is closed."
5JoshuaZ
Yeah, those would be good things to keep up in general. They signal careful planning and good design. And it helps for families who are planning out their summers for the next year or something similar. We don't lose anything by having that data.
3beoShaffer
Just keep in mind that having application information available can imply that applications are still open. So make it clear that the info is just for reference.
[-]Incorrect10

Do you have any direct advice to young programmers?

Reply
4lukeprog
Advice toward what goal(s)? Reducing AI risk?
4Incorrect
Becoming involved with the SI and knowing if they are qualified to be involved with the SI and if not, becoming qualified to be involved with the SI.
5lukeprog
Well, we need lots of help besides elite young math/compsci talent. You could contact louie.helm [at] singinst.org and explain your experience and qualifications. Thanks for your interest!
1John_Maxwell
Is it really optimal to dismiss Incorrect as not being elite math/computer science talent so quickly? Also, are you familiar with growth versus static models of intelligence? This looks to me like you are promoting a static model, which amounts to destroying a public good in my view. University professors don't tell students they are too stupid to contribute to the problems they are trying to solve. I don't see why SI should either.
7PECOS-9
I didn't interpret lukeprog's comment as dismissing Incorrect as not being elite talent. I thought he was just noting that, whether he is "elite" or not, he can contact Louie to find out how he can help.
9lukeprog
Correct.
2JoshuaZ
While I agree with most of this (and have upvoted) two points stand out: I don't think bringing this up helps your point very much. While there are individuals whose apparent extreme talent blooms fairly late (e.g. Steven Chu who didn't really start being that impressive until he was in college), the lack of change of IQ scores over time on average is very robust, dating back to Spearman's original research about a hundred years ago. This is also true for other metrics of intelligence. By and large, intelligence is pretty static. This is true, but professors do sometimes tell students when a problem may just be out of their league. To use an extreme example, consider a grad student who walks into his adviser's office and says he wants to prove the Riemann Hypothesis. That said, your essential point is valid, because even in that case, a professor could still direct them to some easier related problem or helpful question related to some aspect of it. So your basic point is valid.
[-]Kaj_Sotala140

Intelligence seems relatively static, but AFAIK once you've reached a certain minimum threshold in intelligence, conscientiousness becomes a more important factor for actual accomplishment. (Anecdotally and intuitively, conscientiousness seems more amenable to change, but I don't know if the psychological evidence supports that.)

Reply
6Barry_Cotter
Wait, there's real evidence of durable changes in conscientiousness? Point me its way. The psychology literature does not appear (after a brief search) to support the idea of lasting change. I would be happy to be wrong.
8gwern
Well, there's http://commonsenseatheism.com/wp-content/uploads/2011/02/Eisenberger-Learned-industriousness.pdf
4Kaj_Sotala
Sorry, I should have been more clear: I only have anecdotal evidence, and a rather small sample at that. I'll edit my comment.
0John_Maxwell
Mind sharing your source for relatively static IQ? I feel like I've read otherwise, especially for children.
5[anonymous]
Childhood IQs don't correlate that tightly with adult IQs. But once people are in their late teens change already becomes very unlikely.
1JoshuaZ
Yes, in the lower end there's some flexbility, especially in the mid teens but after that change is relatively static.
0faul_sname
I'm not sure how strongly IQ correlates with real-world abilities (well, actually, I am sure: 0.2-0.6 depending on the task 1). You don't need exceptional IQ to do new math (see Richard Feynman) but you do need an interest in math and quite a bit of exposure. Synesthesia can also be helpful. I'm not finding a non-paywalled version right now, and unfortunately am not at my university at the moment to access it.
4John_Maxwell
How many mathematicians consciously try to extract heuristics from their problem-solving process and keep them in a database, or track how environmental factors like diet and activities affect their productivity? Has there ever been a team of mathematicians teamed with the team of mathematician optimizers who observed the mathematicians like lab animals? :D
7gwern
Soviet Russia produced a remarkable amount of math, and ideologically was well-suited to such testing or design; they ultimately created whole academic cities for science and math, optimized (or at least, not pessimized like the rest of Soviet Russia) for research. In fact, what I know of the Russian math academic system strikes me as reminiscent of the impression I have of the very successful athletic systems in both Russia and America: take young kids showing promise with relatives in related areas, push them hard with experienced tutors themselves skilled in the area, provide the resources they might need, various incentives for them and the relatives, and don't let off the slack until they begin to flag in their late 20s/early 30s at which point they take their tutors' places.
0gwern
Read this today, "Rethinking Giftedness and Gifted Education: A Proposed Direction Forward Based on Psychological Science", which is very germane to this discussion. It also discusses athletics.
2private_messaging
I studied in specialized soviet school (well, post soviet, but same teachers). It had tough entrance exam. I say in past tense because it was dismantled. The biggest thing about those is that we study deeper and with better understanding instead of skipping ahead to make prodigies that understand same topics equally badly but at earlier age, and never really become very competent at anything. Also, on the humanities, while there may be less % of humanities, the students are smarter and go ahead faster and still retain/understand more than average at typical humanities course.
3faul_sname
Did you just go meta on the process of going less meta?
3John_Maxwell
A syllabus of recommended reading for folks who think they might want to work on FAI could potentially have a really high benefit to cost ratio. Could potentially have just as high a net benefit for reaching young talent as SPARC. Wouldn't necessarily take too much effort either, maybe just EY spending an hour brainstorming books an ideal collaborator would have read, and setting up a google group for people working through the syllabus. I guess this could potentially increase UFAI risk a little, but I still judge it to be positive expectation. (SPARC could potentially increase UFAI risk too.)
7lukeprog
Already done.
[-]John_Maxwell130

Another point: I seem to recall a joke among mathematicians that if only it was announced that some famous problem was solved, without there actually being a solution, someone would try to find the solution for themselves and succeed in finding a valid solution.

In other words, how problems are framed may be important, and framing a problem as potentially impossible may make it difficult for folks to solve it.

Additionally, I see little evidence that the problems required for FAI are actually hard problems. This isn't to say that it's not a major research endeavor, which it may or may not be. All I'm saying is I don't see top academics having hammered at problems involved in building a FAI the same way they've hammered at, say, proving the Riemann hypothesis.

EY thinking they are super hard doesn't seem like much evidence to me; he's primarily known as a figure in the transhumanist movement and for popular writings on rationality, not for solving research problems. It's not even clear how much time he's spent thinking about the problems in between all of the other stuff he does.

FAI might just require lots of legwork on problems that are relatively straightforward to solve, really.

Reply
8Vladimir_Nesov
IMO the extent to which some/most of these books/documents are only tentative suggestions with unclear relevance to the problem should be emphasized, for example they shouldn't be referred to with "After learning these basics", as if the list is definitive and works as some sort of prerequisite. Also, using the words "deep understanding of mathematics, logic, and computation" to refer to the section with Sipser's introductory text is not really appropriate.
6Will_Newsome
That's cool and a good intro, but you could also have a list of weaker suggestions over ten times that size to show people what sorts of advanced maths &c. might or might not end up being relevant. E.g., a summary paper from the literature on abstract machines, or even extremely young, developing subfields such as quantum algorithmic information theory that teach relevant cognitive-mathematical skills even if they're not quite fundamental to decision theory. This is also a sly way to interest people from diverse advanced disciplines. Is opportunity cost the reason such a list isn't around? My apologies if this question is missing the point of the discussion, and I'm sorry it's only somewhat related to the post, which is an important topic itself.
6John_Maxwell
Nice! That list doesn't actually seem very intimidating; for some reason I expected more highly technical AI papers and books. Why do you guys feel you need elite math talent as opposed to typical math grad student level talent? Which problems, if any, related to FAI seem unusually difficult compared to typical math research problems?
4lukeprog
Now we've come to the point where I'd like to be able to hand you Open Problems in Friendly AI, but I can't.
5John_Maxwell
In the Singularity Institute open problems document, you write: Are you sure raw math talent is the best predictor of a person's ability to do this? I tend to associate this skill with programming especially, and maybe solving math word problems.
1lukeprog
No, I'm not sure. The raw math talent thing is aimed more at the "Eliezer-led basement FAI team" stage.
[-]John_Maxwell130

Does Eliezer have experience with managing research teams?

Reply
6lukeprog
No. I should have said "Eliezer-guided," or something. Eliezer doesn't think it's a good idea for him to manage the team. We need our "Oppenheimer" for that.
Moderation Log
75Comments