Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Q&A with new Executive Director of Singularity Institute

26 Post author: lukeprog 07 November 2011 04:58AM

Today I was appointed the new Executive Director of Singularity Institute.

Because I care about transparency, one of my first projects as an intern was to begin work on the organization's first Strategic Plan. I researched how to write a strategic plan, tracked down the strategic plans of similar organizations, and met with each staff member, progressively iterating the document until it was something everyone could get behind.

I quickly learned why there isn't more of this kind of thing: transparency is a lot of work! 100+ hours of work later, plus dozens of hours from others, and the strategic plan was finally finished and ratified by the board. It doesn't accomplish much by itself, but it's one important stepping stone in building an organization that is more productive, more trusted, and more likely to help solve the world's biggest problems.

I spent two months as a researcher, and was then appointed Executive Director.

In further pursuit of transparency, I'd like to answer (on video) submitted questions from the Less Wrong community just as Eliezer did two years ago.

 

The Rules

1) One question per comment (to allow voting to carry more information about people's preferences).

2) Try to be as clear and concise as possible. If your question can't be condensed into one paragraph, you should probably ask in a separate post. Make sure you have an actual question somewhere in there (you can bold it to make it easier to scan).

3) I will generally answer the top-voted questions, but will skip some of them. I will tend to select questions about Singularity Institute as an organization, not about the technical details of some bit of research. You can read some of the details of the Friendly AI research program in my interview with Michael Anissimov.

4) If you reference certain things that are online in your question, provide a link.

5) This thread will be open to questions and votes for 7 days, at which time I will decide which questions to begin recording video responses for.

 

I might respond to certain questions within the comments thread and not on video; for example, when there is a one-word answer.

Comments (177)

Comment author: XiXiDu 07 November 2011 10:48:33AM 41 points [-]

What would the SIAI do given various amounts of money? Would it make a difference if you had 10 or 100 million dollars at your disposal, would a lot of money alter your strategic plan significantly?

Comment author: ahartell 07 November 2011 05:24:21AM *  34 points [-]

In general, what will you be doing as Executive Director?

(This might be a question you could answer briefly as a reply to this comment.)

Comment author: betterthanwell 07 November 2011 06:10:42AM 22 points [-]

And how will your duties differ from those of the President.

Comment author: Daniel_Burfoot 08 November 2011 12:44:54AM 29 points [-]

Congratulations, but why do you think your comparative advantage lies in being an executive director? Won't that cut into your time budget for reading, writing, and thinking?

Comment author: orthonormal 07 November 2011 05:36:18AM 29 points [-]

To the extent that SIAI intends to work directly on FAI, potential donors (and many others) need to evaluate not only whether the organization is competent, but whether it is completely dedicated to its explicitly altruistic goals.

What is SIAI doing to ensure that it is transparently trustworthy for the task it proposes?

(I'm more interested in structural initiatives than in arguments that it'd be silly to be selfish about Singularity-sized projects; those arguments are contingent on SIAI's presuppositions, and the kind of trustworthiness I'm asking about encompasses the veracity of SIAI on these assumptions.)

Comment author: gwern 07 November 2011 08:00:22PM 21 points [-]

For example, have we heard anything about that big embezzlement?

Comment author: lukeprog 09 November 2011 08:24:51PM 10 points [-]

Some of the money has been recovered. The court date that concerns most of the money is currently scheduled for January 2012.

Comment author: Giles 03 March 2012 04:55:50PM 3 points [-]

January 2012 has passed; any update?

Comment author: lukeprog 03 March 2012 08:17:11PM 5 points [-]

As I understand it, we won a stipulated judgment for repayment of $40k+ of it. Another court date has been scheduled (I think for late March?) to give us a chance to argue for the rest of what we're owed.

Comment author: Gastogh 10 May 2012 09:16:43AM 3 points [-]

Late March has passed. How did things pan out?

Comment author: lukeprog 10 May 2012 03:57:50PM 5 points [-]

We won some more repayment in another stipulated judgment and there's another court date this month.

Comment author: VNKKET 08 November 2011 04:39:20AM *  5 points [-]

Good question. And for people who missed it, this refers to money that was reported stolen on SI's tax documents a few years ago. (relevant thread)

Comment author: lukeprog 09 November 2011 08:25:35PM 1 point [-]

I'm more interested in structural initiatives

Can you give any examples of what you're thinking of, so I can be clearer about what you have in mind when you ask your question?

Comment author: orthonormal 10 November 2011 12:25:13AM 8 points [-]

I'm actually not coming up with any- it seems to be a tough problem. Here's an elaborate hypothetical that I'm not particularly worried about, but which serves as a case study:

Suppose that Robin Hanson is right about the Singularity (no discontinuity, no singleton, just rapid economic doubling until technology reaches physical limits, at which point it's a hardscrapple expansion through the future lightcone for those rich enough to afford descendants), and that furthermore, EY knows it and has been trying to deceive the rest of us in order to fund an early AI, and thus grab a share of the Singularity pie for himself and a few chosen friends.

The thing that makes this seem implausible right now are that the SIAI people I know don't seem to be the sort of people who are into long cons, and also, their object-level arguments about the Singularity make sense to me. But, uh, I'm not sure that I can stake the future on my ability to play a game of Mafia. So I'm wondering if SIAI has come up with any ideas (stronger than a mission statement) to make credible their dedication to a fair Singularity.

Comment author: lukeprog 10 November 2011 02:25:31AM 4 points [-]

Right.

I haven't devoted much time to this because I don't think anybody who has ever interacted with us in person has ever thought this was likely, and I'm not sure if anyone even on the internet has ever made the accusation - though of course some have raised the vague possibility, as you have. In other words, I doubt this worry is anyone's true rejection, whereas I suspect the lack of peer-reviewed papers from SIAI is many people's true rejection.

Comment author: orthonormal 10 November 2011 06:08:50PM 6 points [-]

Skepticism about SIAI's competence screens off skepticism about SIAI's intentions, so of course that's not the true rejection for the vast majority of people. But it genuinely troubles me if nobody's thought of the latter question at all, beyond "Trust us, we have no incentive to implement anything but CEV".

If I told you that a large government or corporation was working hard on AGI plus Friendliness content (and that they were avoiding the obvious traps), even if they claimed altruistic goals, wouldn't you worry a bit about their real plan? What features would make you more or less worried?

Comment author: Vladimir_Nesov 10 November 2011 09:43:49PM *  1 point [-]

I think the key point is that we're not there yet. Whatever theoretical tools we shape now are either generally useful, or generally useless, irrespective of considerations of motive; currently relevant question is (potential) competence. Only at some point in the (moderately distant) future, conditional on current and future work bearing fruit, motive might become relevant.

Comment author: hairyfigment 23 November 2011 09:26:15PM 0 points [-]

What features would make you more or less worried?

I'd worry about selfish institutional behavior, or explicit identification of the programmers' goals with the nation/corporation's selfish interests. Also, I guess, belief in the moral infallibility of some guru.

Otherwise I wouldn't worry about motives, not unless I thought one programmer could feasibly deceive the others and tell the AI to look only at this person's goals. Well, I have to qualify that -- if everyone in the relevant subculture agreed on moral issues and we never saw any public disagreement on what the future of humanity should look like, then maybe I'd worry. That might give each of them a greater expectation of getting what they want if they go with a more limited goal than CEV.

Comment author: Giles 03 March 2012 05:26:58PM 0 points [-]

An "outside view" might be to put the SI in the reference class of "groups who are trying to create a utopia" and observe that previous such efforts that have managed to gain momentum have tended to make the world worse.

I think the reality is more complicated than that, but that might be part of what motivates these kind of questions.

I think the biggest specific trust-related issue I have is with CEV - getting the utility function generation process right is really important, and in an optimal world I'd expect to see CEV subjected to a process of continual improvement and informed discussion. I haven't seen that, but it's hard to tell whether the SI are being overly protective of their CEV document or whether it's just really hard getting the right people talking about it in the right way.

Comment author: wedrifid 10 November 2011 09:32:18AM 0 points [-]

Am I to take this as a general answer to the overall question of trustworthiness or is this intended just as an answer to the specific example?

Comment author: wedrifid 10 November 2011 09:25:55AM *  2 points [-]

Suppose that Robin Hanson is right about the Singularity (no discontinuity, no singleton, just rapid economic doubling until technology reaches physical limits, at which point it's a hardscrapple expansion through the future lightcone for those rich enough to afford descendants), and that furthermore, EY knows it and has been trying to deceive the rest of us in order to fund an early AI, and thus grab a share of the Singularity pie for himself and a few chosen friends.

It would be clearer to say that Robin is right about the future, that there will not be a singularity. A hardscrapple race through the frontier basically just isn't one.

Comment author: jimrandomh 31 December 2011 08:55:15AM 0 points [-]

If you want to hypothesize that SingInst has secrets plus an evil plan, the secrets and plan have to combine in such a way that it's a good plan.

Comment author: XiXiDu 07 November 2011 11:04:10AM 68 points [-]

If someone as capable as Terence Tao approached the SIAI, asking if they could work full-time and for free on friendly AI, what would you tell them to do? In other words, are there any known FAI sub-problems that demand some sort of expertise that the SIAI is currently lacking?

Comment author: [deleted] 05 June 2013 09:38:19AM 1 point [-]

What message about FAI/MIRI should I take away from the fact that this very important question isn't answered?

Comment author: Wei_Dai 08 November 2011 09:23:05AM 19 points [-]

There have been several questions about transparency and trust. In that vein, is there any reason not to publish the minutes of SIAI's board meetings?

Comment author: Wei_Dai 08 November 2011 09:22:50AM 18 points [-]

Much of SIAI's research (Carl Shulman's in particular) are focused not directly on FAI but more generally on better understanding the dynamics of various scenarios that could lead to a Singularity. Such research could help us realize a positive Singularity through means other than directly building an FAI.

Does SIAI have any plans to expand such research activities, either in house, or by academia or independent researchers? (If not, why?)

Comment author: Kaj_Sotala 08 November 2011 08:03:01AM 16 points [-]

SI has traditionally been doing more outreach than actual research. To what extent will the organization be concentrating on research and to what extent will it be concentrating on outreach in the future?

Comment author: JoshuaZ 07 November 2011 06:24:47AM 16 points [-]

Less Wrong is run in cooperation by the SIAI and the FHI (although in practice neither seems to have much day-to-day impact). In your view, how should the SIAI and the FHI interact and what sort of joint projects (if any) should they be doing? Do they share complementary or overlapping goals?

Comment author: JoshuaZ 07 November 2011 06:21:57AM 15 points [-]

The SIAI runs the Singularity Summits. These events have generally been successful, getting a large number of interdiscplinary talks with interesting speakers. However, very little of that work seems to be connected to the SI's longterm goals. In your view, should the summits be more narrowly tailored to the interests of the SI?

Comment author: CarlShulman 11 November 2011 04:02:32AM 4 points [-]

It's actually rather hard to fill the roster with people who have much new and interesting to say on core issues. At the present margin my sense is that this is limited on the supply side.

Comment author: JoshuaZ 11 November 2011 04:06:40AM 0 points [-]

That's an interesting claim. Is there really a tiny set that has new and interesting things to say or is that that set intersected with the set of willing speakers is small? The first is surprising and disturbing. The second seems much less so.

Comment author: CarlShulman 11 November 2011 04:22:46AM *  4 points [-]

There are very few folk who are working on the topic as such, or have written something substantial about it, and a large fraction of those have already spoken. Maybe you could name 10 candidates to give a sense of who you're thinking of? Speakers are already being sought for next year's Summit and good suggestions are welcome.

Some folk are hard to get in any given year because of their packed schedules or other barriers, even though we would want them as speakers (e.g. Bill Joy, various academics) although this becomes easier with time as people like Peter Norvig, Rodney Brooks, Jaan Taallinn, Justin Rattner, etc speak. Others have some interesting things to say, but are just too low-profile relative to the expected value of their talks (such that if SI accepted all such people the Summit's reputation and attendance would be unsustainable). Or, they may just be "in the closet" so that we have no way to locate them as folk with new non-public insights on core issues.

Comment author: JoshuaZ 11 November 2011 04:25:53AM 0 points [-]

I was thinking for example Scott Aaronson if you could get him to give a talk. I'd be interested in for example what he would have to say about theoretical computer science being relevant for AI undergoing fast recursive self-improvement. He's also wrote more generally about philosophical issues connecting to computational complexity some of which might be more directly relevant to Friendly AI.

Comment author: CarlShulman 11 November 2011 04:35:21AM 2 points [-]

Folk around here talk to Scott reasonably often. In my experience, he hasn't been that interested in the core issues you were talking about. A generic tour of computational complexity theory would seem to go in the same category as other relatively peripheral talks e.g. on quantum computing or neuroimaging technology. You're right that the philosophy and computer science stuff he has been doing recently might naturally lend itself to a more "core" talk.

Any others?

Comment author: JoshuaZ 11 November 2011 04:57:18AM 0 points [-]

Not that immediately comes to mind, no.

Comment author: wedrifid 07 November 2011 06:13:52AM *  40 points [-]

The staff and leadership at the SIAI seems to be undergoing a lot of changes recently. Is instability in the organisation something to be concerned about?

Comment author: XiXiDu 07 November 2011 10:43:04AM 37 points [-]

What is each member of the SIAI currently doing and how is it related to friendly AI research?

Comment author: lukeprog 11 November 2011 08:25:04PM 3 points [-]

The Team page can answer much of this question. Is there any staff member in particular for whom the connection between their duties and our mission is unclear?

(Carl isn't on the page yet; we need to get his photo.)

Comment author: XiXiDu 12 November 2011 10:15:22AM *  15 points [-]

The Team page can answer much of this question. Is there any staff member in particular for whom the connection between their duties and our mission is unclear?

Louie Helm is Singularity Institute's Director of Development. He manages donor relations, grant writing, and talent recruitment.

Here are some of the actions that I would take as a director of development:

  • Talk to Peter Thiel and ask him why he donated more money to the Seasteading Institute than the SIAI.
  • Sit down with other SIAI members and ask what talents we need so I can actually get in touch with them.
  • Visit various conferences and ask experts how they would use their expertise if they were told to ensure the safety of artificial general intelligence.

Michael Anissimov is responsible for compiling, distributing, and promoting SIAI media materials.

What I would do:

  • Ask actual media experts what they would do, like those who created the creationist viral video Expelled or the trailer for the book You Are Not So Smart.
  • Talk to Kurzweil if he would be willing to concentrate more strongly on the negative effects of a possible Singularity and promote the Singularity Institute.
  • I would ask Peter Thiel and Jaan Tallinn if they could actually use their influence or companies to promote the Singularity Institute.
  • Talk with other members about the importance of public relations and teach them how to deal with the media.

Anna Salamon is a full-time SIAI researcher.

What is she researching right now? With due respect, but the Uncertain Future web project doesn't look like something that a researcher, who is capable of making progress on the FAI problem, could work 3 years on.

Eliezer Yudkowsky is the foremost researcher on Friendly AI and recursive self-improvement.

He's still writing his book on rationality? How is it going? Is he planning a book tour? Does he already know who he is going to send the book for free, e.g. Richards Dawkins or other people who could promote it on their blog?

Edwin Evans is the Chairman of the Singularity Institute Board of Directors

No clue what he is, or could be doing right now.

Ray Kurzweil

It looks like he's doing nothing except being part of the team page.

Amy Willey, J.D., is the Singularity Institute's Chief Operating Officer, and is responsible for institute operations and legal matters.

What I would do:

  • Try to figure out and make a detailed plan on how to stop possible dangerous AGI projects by all legal means (there are various researchers who believe that superintelligence could happen before 2030).
  • Devise a plan on how to deal with legal challenges arising from possible terrorist attacks done by people who loosely associated themselves with the mission of the SIAI, without its knowledge. For example how to deal with a house search.

Michael Vassar is SIAI's President, and provides overall leadership of the SIAI

As a president, one of the first actions I would take is to talk with everyone about the importance of data security. I would further make sure that there are encrypted backups, of my organisations work, on different continents and under different jurisdictions to make sure that various kinds of catastrophes, including a obligation to disclosure by a government, can be mitigated or avoided.

Comment author: JoshuaZ 07 November 2011 05:57:14AM *  35 points [-]

A lot of Eliezer's work has been not at all related strongly to FAI but has been to popularizing rational thinking. In your view, should the SIAI focus exclusively on AI issues or should it also care about rational issues? In that context, how does Eliezer's ongoing work relate to the SIAI?

Comment author: JoshuaZ 07 November 2011 06:02:40AM *  31 points [-]

One serious danger for organizations is that they can easily outlive their usefulness or can convince themselves that they are still relevant when they are not. Essentially this is a form of lost purpose. This is not a bad thing if the organizations are still doing useful work, but this isn't always the case. In this context, are there specific sets of events (other than the advent of a Singularity) which you think will make the SIAI need to essentially reevaluate its goals and purpose at a fundamental level?

Comment author: Bruno_Coelho 07 November 2011 08:48:57PM 10 points [-]

The SIAI is planning to publish more on academic journals?

Comment author: JoshuaZ 07 November 2011 05:48:47AM *  10 points [-]

In a previous essay, you talked about the optimizer's curse being relevant for calculating utility in the context of existential risk. In that thread, I asked if you had actually gone and applied the method in question to the SIAI. Have you done so yet and if so, what did you find?

Comment author: lukeprog 07 November 2011 06:12:40AM 4 points [-]

No, we have not.

Comment author: beoShaffer 07 November 2011 07:51:09PM 4 points [-]

Do you intend to so in the future, if so then when?

Comment author: XiXiDu 07 November 2011 10:34:39AM 22 points [-]

In June you indicated that exciting developments are happening right now but that it will take a while for things to happen and be announced. Are those developments still in progress?

Comment author: lukeprog 09 November 2011 08:39:34PM 7 points [-]

I'll answer this one here.

My comment in June was in response to Normal_Anomaly's comment:

Count me as another person who would switch some of my charitable contribution from VillageReach to SIAI if I had more information on this subject [what research will be done with donated funds].

I replied:

the most exciting developments in this space in years (to my knowledge) are happening right now, but it will take a while for things to happen and be announced.

To my memory, I had two things in mind:

  • The Strategic Plan I was then developing, which does a better job of communicating what SIAI will do with donated funds than ever before. This was indeed board-ratified and published.
  • A greater push from SIAI to publish its research.

The second one takes longer but is in progress. We do have several chapters forthcoming in The Singularity Hypothesis volume from Springer, as well as other papers in the works. We have also been actively trying to hire more researchers. I was the first such hire, and have 1-4 papers/chapters on the way, but am now Executive Director. We tried to hire a few other researchers, but they did not work out. Recruiting researchers to work on these problems has been difficult for both SIAI and FHI, but we continue to try.

Mostly, we need (1) more funds, and (2) smart people who not only say they think AI risk is the most important problem in the world, but who are willing to make large life changes as if those words reflect their actual anticipations. (Of course I don't mean that the rational thing to do if you're a smart researcher who cares about AI risk is to come work for Singularity Institute, but that should be true for some smart researchers.)

Comment author: VincentYu 10 November 2011 10:59:54PM 1 point [-]

[people] who are willing to make large life changes

What sort of life changes?

Comment author: lukeprog 10 November 2011 11:24:07PM *  5 points [-]

For example, moving to the Bay Area to be paid to do research on particular sub-problems of Friendly AI research.

Or at the very least, doing some of these small tasks.

Comment author: Xom 07 November 2011 05:54:56AM *  22 points [-]

What is your information diet like? (I mean other than when you engage in focused learning.) Do you regulate it, or do you just let it happen naturally?

By that I mean things like:

  • Do you have a reading schedule (e.g. X hours daily)?
  • Do you follow the news, or try to avoid information with a short shelf-life?
  • Do you significantly limit yourself with certain materials (e.g. fun stuff) to focus on higher priorities?
  • In the end, what is the makeup of the diet?
  • Etc.

Inspired by this question (Eliezer's answer).

Comment author: lukeprog 09 November 2011 03:02:00AM 8 points [-]

This is not much about Singularity Institute as an organization, so I'll just answer it here in the comments.

  • I do not regulate my information diet.
  • I do not have a reading schedule.
  • I do not follow the news.
  • I haven't read fiction in years. This is not because I'm avoiding "fun stuff," but because my brain complains when I'm reading fiction. I can't even read HPMOR. I don't need to consciously "limit" my consumption of "fun stuff" because reading scientific review articles on subjects I'm researching and writing about is the fun stuff.
  • What I'm trying to learn at this moment almost entirely dictates my reading habits.
  • The only thing beyond this scope is my RSS feed, which I skim through in about 15 minutes per day.
Comment author: Aleksei_Riikonen 10 November 2011 02:55:56AM 0 points [-]

I'm glad to hear I'm not the only fan of Eliezer who isn't reading HPMOR.

In general, like you I also don't tend to get any fiction read (unlike earlier). For years, I haven't progressed on several books I've got started that I enjoy reading and consider very smart also in a semi-useful way. It's rather weird really, since simultaneously I do with great enthusiasm watch some fictional movies and tv series, even repeatedly. (And I do read a considerable amount of non-fiction.)

And I follow the news. A lot. The number one fun thing for me, it seems.

Comment author: quartz 07 November 2011 07:19:21AM 65 points [-]

How are you going to address the perceived and actual lack of rigor associated with SIAI?

There are essentially no academics who believe that high-quality research is happening at the Singularity Institute. This is likely to pose problems for your plan to work with professors to find research candidates. It is also likely to be an indicator of little high-quality work happening at the Institute.

In his recent Summit presentation, Eliezer states that "most things you need to know to build Friendly AI are rigorous understanding of AGI rather than Friendly parts per se". This suggests that researchers in AI and machine learning should be able to appreciate high-quality work done by SIAI. However, this is not happening, and the publications listed on the SIAI page--including TDT--are mostly high-level arguments that don't meet this standard. How do you plan to change this?

Comment author: CarlShulman 11 November 2011 03:52:55AM *  18 points [-]

There are essentially no academics who believe that high-quality research is happening at the Singularity Institute.

David Chalmers has said that the decision theory work is a major advance (along with various other philosophers), although he is frustrated that it hasn't been communicated more actively to the academic decision theory and philosophy communities. A number of current and former academics, including David, Stephen Omohundro, James Miller (above), and Nick Bostrom have reported that work at SIAI has been very helpful for their own research and writing in related topics.

Evan Williams, now a professor of philosophy at Purdue cites, in his dissertation, three inspirations leading to the work: John Stuart Mill's "On Liberty," John Rawls' "Theory of Justice," and Eliezer Yudkowsky's "Creating Friendly AI" (2001), discussed at greater length than the others. Nick Beckstead, a Rutgers (#2 philosophy program) philosophy PhD student who works on existential risks and population ethics reported large benefits to his academic work from discussions with SIAI staff.

These folk are a minority, and SIAI is not well integrated with academia (no PhDs on staff, publishing, etc), but also not negligible.

In his recent Summit presentation, Eliezer states that "most things you need to know to build Friendly AI are rigorous understanding of AGI rather than Friendly parts per se". This suggests that researchers in AI and machine learning should be able to appreciate high-quality work done by SIAI.

I think that work in this area has been disproportionately done by Eliezer Yudkowsky, and to a lesser extent Marcello Herreshoff. Eliezer has been heavily occupied with Overcoming BIas, Less Wrong, and his book for the last several years, in part to recruit a more substantial team for this. He also is reluctant to release work that he thinks is relevant to building AGI. Problems in recruiting and the policies of secrecy seem like the big issues here.

Comment author: Wei_Dai 13 November 2011 11:34:56AM 13 points [-]

Eliezer has been heavily occupied with Overcoming BIas, Less Wrong, and his book for the last several years, in part to recruit a more substantial team for this.

Eliezer's investment into OB/LW apparently hasn't returned even a single full-time FAI researcher for SIAI after several years (although a few people are almost certainly doing more and better FAI-related research than if the Sequences didn't happen). Has this met SIAI's initial expectations? Do you guys think we're at the beginning of a snowball effect, or has OB/LW pretty much done as much as it can, as far as creating/recruiting FAI researchers is concerned? What are your current expectations for the book in this regard?

Comment author: CarlShulman 13 November 2011 08:24:34PM *  11 points [-]

I have noticed increasing numbers of very talented math and CS folk expressing interest or taking actions showing significant commitment. A number of them are currently doing things like PhD programs in AI. However, there hasn't been much of a core FAI team and research program to assimilate people into. Current plans are for Eliezer to switch back to full time AI after his book, with intake of more folk into that research program. Given the mix of people in the extended SIAI community, I am pretty confident that with abundant funding a team of pretty competent researchers (with at least some indicators like PhDs from the top AI/CS programs, 1 in 100,000 or better performance on mathematics contests, etc) could be mustered over time, based on people I already know.

I am less confident that a team can be assembled with so much world-class talent that it is a large fraction of the quality-adjusted human capital applied to AGI, without big gains in recruiting (e.g. success with the rationality book or communication on AI safety issues, better staff to drive recruiting, a more attractive and established team to integrate newcomers, relevant celebrity endorsements, etc). The Manhattan Project had 21 then- or future Nobel laureates. AI, and certainly FAI, are currently getting a much, much smaller share of world scientific talent than nukes did, so that it's easier for a small team to loom large, but it seems to me like there is still a lot of ground to be covered to recruit a credibly strong FAI team.

Comment author: Wei_Dai 14 November 2011 07:49:28AM 7 points [-]

Thanks. You didn't answer my questions directly, but it sounds like things are proceeding more or less according to expectations. I have a couple of followup questions.

At what level of talent do you think an attempt to build an FAI would start to do more (expected) good than harm? For simplicity, feel free to ignore the opportunity cost of spending financial and human resources on this project, and just consider the potential direct harmful effects, like accidentally creating an UFAI while experimenting to better understand AGI, or building a would-be FAI that turns out to be an UFAI due to a philosophical, theoretical or programming error, or leaking AGI advances that will allow others to build an UFAI, or starting an AGI arms race.

I have a serious concern that if SIAI ever manages to obtain abundant funding and a team of "pretty competent researchers" (or even "world-class talent", since I'm not convinced that even a team of world-class talent trying to build an FAI will do more good than harm), it will proceed with an FAI project without adequate analysis of the costs and benefits of doing so, or without continuously reevaluating the decision in light of new information. Do you think this concern is reasonable?

If so, I think it would help a lot if SIAI got into the habit of making its strategic thinking more transparent. It could post answers to questions like the ones I asked in the grandparent comment without having to be prompted. It could publish the reasons behind every major strategic decision, and the metrics it keeps to evaluate its initiatives. (One way to do this, if such strategic thinking often occurs or is presented at board meetings, would be to publish the meeting minutes, as I suggested in another comment.)

Comment author: CarlShulman 14 November 2011 09:18:47AM *  4 points [-]

At what level of talent do you think an attempt to build an FAI would start to do more (expected) good than harm?

I'm not sure that scientific talent is the relevant variable here. More talented folk are more likely to achieve both positive and negative outcomes. I would place more weight on epistemic rationality, motivations (personality, background checks), institutional setup and culture, the strategy of first trying to get test the tractability of robust FAI theory and then advancing FAI before code (with emphasis on the more-FAI-less-AGI problems first), and similar variables.

Do you think this concern is reasonable?

Certainly it's a reasonable concern from a distance. Folk do try to estimate and reduce the risks you mentioned, and to investigate alternative non-FAI interventions. My personal sense is that these efforts have been reasonable but need to be bolstered along with the FAI research team. If it looks like a credible (to me) team may be assembled my plan would be (and has been) to monitor and influence team composition, culture, and exposure to information. In other words, I'd like to select folk ready to reevaluate as well as to make progress, and to work hard to build that culture as researchers join up.

If so, I think it would help a lot if SIAI got into the habit of making its strategic thinking more transparent.

I can't speak for everyone, but I am happy to see SIAI become more transparent in various ways. The publication of the strategic plan is part of that, and I believe Luke is keen (with encouragement from others) to increase communication and transparency in other ways.

publish the meeting minutes

This one would be a decision for the board, but I'll give my personal take again. Personally, I like the recorded GiveWell meetings and see the virtues of transparency in being more credible to observers, and in providing external incentives. However, I would also worry that signalling issues with a diverse external audience can hinder accurate discussion of important topics, e.g. frank discussions of the strengths and weaknesses of potential Summit speakers, partners, and potential hires that could cause hurt feelings and damage valuable relationships. Because of this problem I would be more wholehearted in supporting other forms of transparency, e.g. more frequent and detailed reporting on activities, financial transparency, the strategic plan, things like Luke's Q&A, etc. But I wouldn't be surprised if this happens too.

Comment author: Wei_Dai 15 November 2011 10:23:37AM *  19 points [-]

I'm not sure that scientific talent is the relevant variable here. More talented folk are more likely to achieve both positive and negative outcomes.

Let's assume that all the other variables are already optimized for to minimize the risk of creating an UFAI. It seems to me that the the relationship between the ability level of the FAI team and probabilities of the possible outcomes must then look something like this:

FAI probability chart

This chart isn't meant to communicate my actual estimates of the probabilities and crossover points, but just the overall shapes of the curves. Do you disagree with them? (If you want to draw your own version, click here and then click on "Modify This Chart".)

Folk do try to estimate and reduce the risks you mentioned, and to investigate alternative non-FAI interventions.

Has anyone posted SIAI's estimates of those risks?

I would also worry that signalling issues with a diverse external audience can hinder accurate discussion of important topics

That seems reasonable, and given that I'm more interested in the "strategic" as opposed to "tactical" reasoning within SIAI, I'd be happy for it to be communicated through some other means.

Comment author: Eliezer_Yudkowsky 18 May 2012 10:22:46PM 7 points [-]

I like this chart.

Comment author: CarlShulman 15 November 2011 07:25:45PM *  6 points [-]

Do you disagree with them?

If we condition on having all other variables optimized, I'd expect a team to adopt very high standards of proof, and recognize limits to its own capabilities, biases, etc. One of the primary purposes of organizing a small FAI team is to create a team that can actually stop and abandon a line of research/design (Eliezer calls this "halt, melt, and catch fire") that cannot be shown to be safe (given limited human ability, incentives and bias). If that works (and it's a separate target in team construction rather than a guarantee, but you specified optimized non-talent variables) then I would expect a big shift of probability from "UFAI" to "null."

Comment author: Wei_Dai 15 November 2011 10:08:25PM *  23 points [-]

What I'm afraid of is that a design will be shown to be safe, and then it turns out that the proof is wrong, or the formalization of the notion of "safety" used by the proof is wrong. This kind of thing happens a lot in cryptography, if you replace "safety" with "security". These mistakes are still occurring today, even after decades of research into how to do such proofs and what the relevant formalizations are. From where I'm sitting, proving an AGI design Friendly seems even more difficult and error-prone than proving a crypto scheme secure, probably by a large margin, and there is no decades of time to refine the proof techniques and formalizations. There's good recent review of the history of provable security, titled Provable Security in the Real World, which might help you understand where I'm coming from.

Comment author: cousin_it 16 November 2011 02:23:16PM *  7 points [-]

Your comment has finally convinced me to study some practical crypto because it seems to have fruitful analogies to FAI. It's especially awesome that one of the references in the linked article is "An Attack Against SSH2 Protocol" by W. Dai.

Comment author: John_Maxwell_IV 23 March 2012 06:51:19AM 3 points [-]

From where I'm sitting, proving an AGI design Friendly seems even more difficult and error-prone than proving a crypto scheme secure, probably by a large margin, and there is no decades of time to refine the proof techniques and formalizations.

Correct me if I'm wrong, but it doesn't seem as though "proofs" of algorithm correctness fail as frequently as "proofs" of cryptosystem unbreakableness.

Where does your intuition that friendliness proofs are on the order of reliability of cryptosystem proofs come from?

Comment author: CarlShulman 15 November 2011 10:25:41PM 1 point [-]

What I'm afraid of is that a design will be shown to be safe, and then it turns out that the proof is wrong, or that the formalization of the notion of "safety" used by the proof is wrong.

Thanks for clarifying.

This kind of thing happens a lot in cryptography,

I agree.

Comment author: XiXiDu 15 November 2011 10:58:55AM 2 points [-]

Could you elaborate on the ability axis. Could you name some people that you perceive to be of world class ability in their field. Could you further explain if you believe that there are people who are sufficiently above that class.

For example, what about Terence Tao? What about the current SIAI team?

Comment author: wedrifid 14 November 2011 03:12:58PM *  9 points [-]

However, I would also worry that signalling issues with a diverse external audience can hinder accurate discussion of important topics

Basically it ensures that all serious discussion and decision making is made prior to any meeting in informal conversations so that the meeting sounds good. Such a record should be considered a work of fiction regardless of whether it is a video transcript or a typed document. (Only to the extent that the subject of the meeting matters - harmless or irrelevant things wouldn't change.)

Because of this problem I would be more wholehearted in supporting other forms of transparency, e.g. more frequent and detailed reporting on activities, financial transparency, the strategic plan, things like Luke's Q&A, etc. But I wouldn't be surprised if this happens too.

That's more like it!

Comment author: lukeprog 01 March 2012 10:04:33PM 2 points [-]

Personally, I like the recorded GiveWell meetings and see the virtues of transparency in being more credible to observers, and in providing external incentives. However, I would also worry that signalling issues with a diverse external audience can hinder accurate discussion of important topics, e.g. frank discussions of the strengths and weaknesses of potential Summit speakers, partners, and potential hires that could cause hurt feelings and damage valuable relationships. Because of this problem I would be more wholehearted in supporting other forms of transparency, e.g. more frequent and detailed reporting on activities, financial transparency, the strategic plan, things like Luke's Q&A, etc. But I wouldn't be surprised if this happens too.

I'll take this opportunity to mention that I'm against publishing SIAI's board meeting minutes. First, for the reasons Carl gave above. Second, because then we'd have to invest a lot of time explaining the logic behind each decision, or else face waves of criticism for decisions that appear arbitrary when one merely publishes the decision and not the argument.

However, I'm definitely making big effort to improve SIAI transparency. Our new website (under development) has a page devoted to transparency, where you'll be able to find our strategic plan, our 990s, and probably other links. I'm also publishing the monthly progress reports, and recently co-wrote 'Intelligence Explosion: Evidence and Import', which for the first time (excepting Chalmers) summarizes many of our key pieces of reasoning with the clarity of mainstream academic form. We're also developing an annual report, and I'm working toward developing some other documents that will make SIAI strategy more transparent. But all this takes time, especially when starting from pretty close to 0 on transparency, and having lots of other problems to fix, too.

Comment author: Wei_Dai 01 March 2012 10:30:36PM 10 points [-]

Second, because then we'd have to invest a lot of time explaining the logic behind each decision, or else face waves of criticism for decisions that appear arbitrary when one merely publishes the decision and not the argument.

Are the arguments not made during the board meetings? Or do you guys talk ahead of time and just formalize the decisions during the board meetings?

In any case, I think you should invest more time explaining the logic behind your decisions, and not just make the decisions themselves more transparent. If publishing board meeting minutes is not the best way to do that, then please think about some other way of doing it. I'll list some of the benefits of doing this, in case you haven't thought of some of them:

  • encourage others to emulate you and think strategically about their own choices
  • allow outsiders to review your strategic thinking and point out possible errors
  • assure donors and potential donors that there is good reasoning behind your strategic decisions
  • improve exchange of strategic ideas between everyone working on existential risk reduction
Comment author: lukeprog 01 March 2012 10:41:44PM 3 points [-]

The arguments are strewn across dozens of conversations in and out of board meetings (mostly out).

As for finding other ways to explain the logic behind our decisions, I agree, and I'm working on it. One qualification I would add, however, is that I predict more benefit to my strategic thinking from one hour with Paul Christiano and one hour with Nick Bostrom than from spending four hours to write up my strategic thinking on subject X and publishing it so that passersby can comment on it. It takes a lot of effort to be so well-informed about these issues that one can offer valuable strategic advice. But for some X we have already spent those many productive hours with Christiano and Bostrom and so on, and it's a good marginal investment to write up our strategic thinking on X.

Comment author: Wei_Dai 02 March 2012 07:28:58AM 8 points [-]

This reminds me a bit of Eliezer's excuse when he was resisting calls for him to publish his TDT ideas on LW:

Unfortunately this "timeless decision theory" would require a long sequence to write up

I suggest you may be similarly overestimating the difficulty of explaining your strategic ideas/problems to a sufficiently large audience to get useful feedback. Why not just explain them the same way that you would explain to Christiano and Bostrom? If some among the LW community don't understand, they can ask questions and others could fill them in.

The decision theory discussions on LW generated significant progress, but perhaps more importantly created a pool of people with strong interest in the topic (some of whom ended up becoming your research associates). Don't you think the same thing could happen with Singularity strategies?

Comment author: lessdazed 14 November 2011 03:06:46PM 1 point [-]

signalling issues with a diverse external audience can hinder accurate discussion

Minutes can be much more general than (video) transcripts.

I would be surprised if the optimal solution isn't a third alternative and is instead total secrecy or manipulable complete transcription.

Comment author: XiXiDu 13 November 2011 03:25:51PM 8 points [-]

Eliezer's investment into OB/LW apparently hasn't returned even a single full-time FAI researcher...

I believe that the SIAI has has been very successful in using OB/LW to not only rise awareness of risks from AI but to lend credence to the idea. From the very beginning I admired that feat.

Eliezer Yudkowsky's homepage is a perfect example of its type. Just imagine he would have concentrated solely on spreading the idea of risks from AI and the necessity of a friendliness theory. Without any background relating to business or an academic degree, to many people he would appear to be yet another crackpot spreading prophecies of doom. But someone who is apparently well-versed in probability theory, who studied cognitive biases and tries to refine the art of rationality? Someone like that can't possible be deluded enough to hold some complex beliefs that are completely unfounded, there must be more to it.

That's probably the biggest public relations stunt in the history of marketing extraordinary ideas.

Comment author: Wei_Dai 13 November 2011 04:27:43PM 5 points [-]

Certainly, by many metrics LW can be considered wildly successful, and my comment wasn't meant to be a criticism of Eliezer or SIAI. But if SIAI was intending to build an FAI using its own team of FAI researchers, then at least so far LW has failed to recruit them any such researchers. I'm trying to figure out if this was the expected outcome, and if not, how updating on it has changed SIAI's plans. (Or to remind them to update in case they forgot to do so.)

Comment author: JoshuaZ 13 November 2011 03:53:47PM *  0 points [-]

Most of your analysis seems right, but the last sentence seems likely to be off. There have been a lot of clever PR stunts in history.

Comment author: XiXiDu 13 November 2011 05:32:06PM *  2 points [-]

There have been a lot of clever PR stunts in history.

Most of them have not been targeting smart and educated nonconformists. Eliezer successfully changed people's mind by installing a way of thinking (a framework of heuristics, concepts and ideas) that is fine-tuned to non-obviously culminate in one inevitable conclusion, that you want to contribute money to his charity because it is rational to do so.

Take a look at the sequences in the light of the Singularity Institute. Even the Quantum Sequence helps to hit a point home that is indispensable to convince people, who would otherwise be skeptical, that it is rational to take risks from AI seriously. The Sequences promulgate that logical implications of general beliefs you already have do not cost you extra probability and that it would be logically rude to demand some knowably unobtainable evidence.

A true masterpiece.

Comment author: Dr_Manhattan 10 December 2011 05:12:11PM 0 points [-]

I have informally been probing smart people I meet whether they're aware of LW. The answers have been surprisingly high number of 'Yes'. I expect this is already making impact on, at the very least, a less risky distribution of funding sources, and probably a good increase in funding once some of them (as many are in startups) will hit paydirt.

Comment author: XiXiDu 13 November 2011 03:01:14PM 4 points [-]

He also is reluctant to release work that he thinks is relevant to building AGI.

Sooner or later he will have to present some results. As the advent of AGI is moving closer people will start to panic and demand hard evidence that the SIAI is worth their money. Even someone who has published a lot of material on rationality and a popular fanfic will run out of credit and people will stop taking his word for it.

Comment author: James_Miller 07 November 2011 03:21:35PM *  40 points [-]

There are essentially no academics who believe that high-quality research is happening at the Singularity Institute.

I believe that high-quality research is happening at the Singularity Institute.

James Miller, Associate Professor of Economics, Smith College.

PhD, University of Chicago.

Comment author: XFrequentist 07 November 2011 10:41:55PM 23 points [-]

To distinguish the above from the statement "I like the Singularity Institute", could you be specific about what research activities you have observed in sufficient detail to confidently describe as "high-quality"?

ETA: Not a hint of sarcasm or snark intended, I'm sincerely curious.

Comment author: James_Miller 08 November 2011 01:25:01AM *  24 points [-]

I'm currently writing a book on the Singularity and have consequently become extremely familiar with the organization's work. I have gone through most of EY's writings and have an extremely high opinion of them. His research on AI plays a big part in my book. I have also been ending my game theory classes with "rationality shorts" in which I present some of EY's material from the sequences.

I also have a high opinion of Carl Shulman's (an SI employee) writings including “How Hard is Artificial Intelligence? The Evolutionary Argument and Observation Selection Effects." (Co-authored with Bostrom) and Shulman's paper on AGI and arms races.

Comment author: Solvent 07 November 2011 07:51:02AM 7 points [-]

Luke discussed this a while back here.

I agree that this is an important question.

Comment author: shminux 07 November 2011 07:35:38AM 5 points [-]

the publications listed on the SIAI page--including TDT--are mostly high-level arguments that don't meet this standard. How do you plan to change this?

This is my favorite of the questions so far.

Comment author: lukeprog 13 November 2011 05:19:34PM 2 points [-]

How are you going to address the perceived and actual lack of rigor associated with SIAI?

A clarifying question. By 'rigor', do you mean the kind of rigor that is required to publish in journals like Risk Analysis or Minds and Machines, or do you mean something else by 'rigor'?

Comment author: quartz 14 November 2011 09:23:34AM 7 points [-]

A clarifying question. By 'rigor', do you mean the kind of rigor that is required to publish in journals like Risk Analysis or Minds and Machines, or do you mean something else by 'rigor'?

I mean the kind of precise, mathematical analysis that would be required to publish at conferences like NIPS or in the Journal of Philosophical Logic. This entails development of technical results that are sufficiently clear and modular that other researchers can use them in their own work. In 15 years, I want to see a textbook on the mathematics of FAI that I can put on my bookshelf next to Pearl's Causality, Sipser's Introduction to the Theory of Computation and MacKay's Information Theory, Inference, and Learning Algorithms. This is not going to happen if research of sufficient quality doesn't start soon.

Comment author: lukeprog 14 November 2011 09:26:16AM 2 points [-]

In 15 years, I want to see a textbook on the mathematics of FAI that I can put on my bookshelf next to Pearl's Causality, Sipser's Introduction to the Theory of Computation and MacKay's Information Theory, Inference, and Learning Algorithms.

My day brightened imagining that!

Thanks for clarifying.

Comment author: quartz 16 November 2011 08:26:45PM 0 points [-]

Addendum: Since the people who upvoted the question were in the same position as you with respect to its interpretation, it would be good to not only address my intended meaning, but all major modes of interpretation.

Comment author: XiXiDu 13 November 2011 06:23:31PM *  3 points [-]

By 'rigor', do you mean the kind of rigor that is required to publish in journals like Risk Analysis or Minds and Machines, or do you mean something else by 'rigor'?

I can't speak for the original questioner, but take for example the latest post by Holden Karnofsky from GiveWell. I would like to see a response by the SIAI that applies the same amount of mathematical rigor to show that it actually is the rational choice from the point of view of charitable giving.

A potential donor might currently get the impression that the SIAI has written a lot of rather colloquial posts on rationality than rigorous papers on the nature of AGI, not to mention friendly AI. In contrast, GiveWell appears to concentrate on their main objective, the evaluation of charities. In doing so they are being strictly technical, an appraoch that introduces a high degree of focus by tabooing colloquial language and thereby reducing ambiguity, while allowing others to review their work.

Some of the currently available papers might, in a less favorably academic context, be viewed as some amount of handwaving mixed with speculations.

Comment author: ArisKatsaris 07 November 2011 12:41:28PM *  52 points [-]

I'd like to answer (on video) submitted questions from the Less Wrong community just as Eliezer did two years ago.

That was the most horribly designed thing I've ever seen anyone do on LessWrong, as I once described here so please, please, no video.

The questions are text. Have your answer on text too, so that we can actually read them -- unless there's some particular question which would actually be enhanced by the usage of video, (e.g. you'd like to show an animated graph or a computer simulation or something)

If there's nothing I can say to convince you against using video, then I beg you to atleast take the time to read my more specific problems in the link above and correct those particular flaws - a single audio that we can atleast play and listen in the background, while we're doing something else, instead of 30 videos that we must individually click. If not that, atleast a clear description of the questions on the same page (AND repeated clearly on the audio itself), so that we can see the questions that interest us, instead of a link to a different page.

But please, just consider text instead. Text has the highest signal-to-noise ratio. We can actually read it in our leisure. We can go back and forth and quote things exactly. TEXT IS NIFTY.

Comment author: curiousepic 07 November 2011 04:50:28PM *  15 points [-]

I disagree completely, as video has value not present in text, and text is easily derived from video. If this has not been done for Eliezer's videos, I volunteer to transcribe them - please let me know.

Comment author: cousin_it 07 November 2011 11:57:27PM *  16 points [-]

I just tried to find a transcript for Eliezer's Q&A and couldn't find one. So I'm taking you up on your offer!

Also, video is easily derived from text and I would actually enjoy watching a SingInst Q&A made with that sort of app :-)

Comment author: curiousepic 08 November 2011 04:35:01AM *  21 points [-]

Looks like you're right. I commit to working on this over the next few weeks. Please check in with me every so often (via comment here would be fine) to gauge my progress and encourage completion.

It's approximately 120 minutes of video; taking a number from wikipedia gives me 150 spoken wpm, divided by my typing wpm gives me about 6 hours, which will be optimistic - let's double it to 12, at let's say an average of 30 mins per day gives me 24 days. Let's see how it goes!

Comment author: cousin_it 09 November 2011 09:46:15AM 3 points [-]

Checking in. Do you have the first 750 words done?

Comment author: curiousepic 09 November 2011 02:02:50PM *  7 points [-]

I have the first four, and six of the shortest answers done, so yes. I had a lot of spare time yesterday so I thought I'd get a head start. Today may be similar.

Comment author: curiousepic 11 November 2011 02:11:56AM 5 points [-]

I am now roughly 60% done. I've been spending more time each day than I anticipated; I have been known to overcompensate for the planning fallacy :)

Comment author: mindspillage 09 November 2011 04:44:30AM 2 points [-]

That's what you consider "easily derived"?

Comment author: curiousepic 09 November 2011 02:00:26PM *  5 points [-]

Relative to manifesting video of the person speaking the answers in a genuine manner after the fact, yes. But point taken, the irony of manually transcribing videos from an AI researcher is not lost on me. I feel somewhat like a monk in the Bayesian monastery.

Comment author: alibaba 11 November 2011 01:43:18AM 0 points [-]

Why not just play the audio to something like the Dragon Dictation app on an iPhone and then go back and proof it?

Comment author: curiousepic 11 November 2011 04:08:12AM *  1 point [-]

I'm skeptical of the time it would save. The app won't work for the length of the videos, but if you're aware of another great, free program, let me know.

Comment author: Nick_Roy 07 November 2011 10:36:21PM 6 points [-]

Non-profit organizations like SI need robust, sustainable resource strategies. Donations and grants are not reliable. According to my university Social Entrepreneurship course, social businesses are the best resource strategy available. The Singularity Summit is a profitable and expanding example of a social business.

My question: is SI planning on creating more social businesses (either related or unrelated to the organization's mission) to address long-term funding needs?

By the way, I appreciate SI working on its transparency. According to my studies, transparency and accountability are also essential to the long-term success of a non-profit organization.

Comment author: XFrequentist 07 November 2011 05:13:40PM 16 points [-]

From the Strategic Plan (pdf):

Strategy #3: Improve the function and capabilities of the organization.

  1. Encourage a new organization to begin rationality instruction similar to what Singularity Institute did in 2011 with Rationality Minicamp and Rationality Boot Camp.

Any news on the status of this new organization, or what specific form its activities would take (short courses, camps, etc)?

Comment author: lukeprog 19 November 2011 07:03:56AM 1 point [-]

Two camps are in different stages of planning. A detailed curriculum and materials are also under development. More details forthcoming, though perhaps not until January.

Comment author: Bugmaster 07 November 2011 10:57:59AM 15 points [-]

The stated goal of SIAI is "to ensure that the creation of smarter-than-human intelligence benefits society". What metric or heuristic do you use in order to determine how much progress you (as an organization) are making toward this goal ? Given this heuristic, can you estimate when your work will be complete ?

Comment author: lukeprog 11 November 2011 03:40:52PM 4 points [-]

What metric or heuristic do you use in order to determine how much progress you (as an organization) are making toward this goal?

There is no such metric for mathematical and philosophical breakthroughs. We're just doing it as quickly as we can given our level of funding.

Comment author: XiXiDu 07 November 2011 11:38:46AM *  20 points [-]

Given the nature of friendly AI research, is the SIAI expecting to use its insights into AGI to develop marketable products, to make money from its research, as to not having to rely on charitable contributions in future?

Here is a quote from Holden Karnofsky:

My reasoning is that it seems to me that if they have unique insights into the problems around AGI, then along the way they ought to be able to develop and publish/market innovations in benign areas, such as speech recognition and language translation programs, which could benefit them greatly both directly (profits) and indirectly (prestige, affiliations) - as well as being a very strong challenge to themselves and goal to hold themselves accountable to, which I think is worth quite a bit in and of itself.

Comment author: kilobug 07 November 2011 10:51:02AM 32 points [-]

Congrats Luke !

Just a form/media comment : I would personally greatly prefer a text Q&A page rather than a video, for many reasons (my understanding of written English is higher than of spoken English, text is easier to re-read or read at your own speed, much less intrusive media that I can for example read during small breaks at work while I can't for video, poor Internet bandwidth at home making downloading video always painful to me, ...).

Comment author: orthonormal 07 November 2011 02:20:40PM 24 points [-]

Better yet, video with transcripts.

Comment author: gwern 07 November 2011 08:04:12PM 10 points [-]

Ditto. (Native English speaker, but hearing-impaired.)

Comment author: Bugmaster 07 November 2011 10:59:32AM 7 points [-]

Agreed, text would be quite useful.

Comment author: Kaj_Sotala 07 November 2011 01:16:50PM 5 points [-]

I second this.

Comment author: betterthanwell 07 November 2011 11:00:01PM *  5 points [-]

Do you regard the hard takeoff scenario as possible, plausible, likely?

Comment author: ChrisHallquist 11 November 2011 02:21:34AM 11 points [-]

What does the Executive Director of the Singularity Institute do?

Comment author: XiXiDu 07 November 2011 12:41:01PM 10 points [-]

What security measures does the SIAI take to ensure that it isn't actually increasing existential risks by allowing key insights to leak, either as a result of espionage or careless handling of information?

Comment author: timtyler 07 November 2011 01:03:52PM 1 point [-]

Leaks usually damage the party doing the leaking. Others benefit - and that's usually desirable from the perspective of the rest of society - since it helps to even out power and wealth. Thus the popularity of WikiLeaks.

Comment author: wedrifid 07 November 2011 01:13:56PM 0 points [-]

Leaks usually damage the party doing the leaking.

With the obvious exceptions being insider trading and selling secrets.

Comment author: timtyler 07 November 2011 01:30:32PM *  1 point [-]

So: I didn't mean to refer to the individual responsible for leaking the information, I meant to refer to the organisation which the information is leaking from.

I am sure there are exceptions. For instance, some "leaks" turn out to be marketing.

Comment author: Gedusa 07 November 2011 03:20:57PM 8 points [-]

What initiatives is the Singularity Institute taking or planning to take to increase it's funding to whatever the optimal level of funding is?

Comment author: lessdazed 07 November 2011 02:05:00PM 7 points [-]

I'd like to answer (on video)

Fuzzy. Sounds like a lost purpose. What ArisKatsaris said. Although it's not impossible that Eliezer was deliberately failing as much as humanly possible as an anti-cult measure.

Comment author: Vaniver 07 November 2011 10:57:15PM 7 points [-]

Although it's not impossible that Eliezer was deliberately failing as much as humanly possible as an anti-cult measure.

Incompetence is generally a safer assumption than intentionality.

Comment author: XiXiDu 07 November 2011 10:37:37AM 15 points [-]

Is the SIAI willing to pursue experimental AI research or does it solely focus on hypothetical aspects?

Comment author: JoshuaZ 07 November 2011 05:49:42AM *  15 points [-]

Are you concerned about potential negative signaling/ status issues that will occur if the SIAI has as an executive director someone who was previously just an intern?

Comment author: MichaelAnissimov 18 November 2011 08:54:17PM *  3 points [-]

As a long-time employee I'd actually say that this is a good thing because it shows that there is a meritocratic structure where new arrivals can rise quickly due to good performance.

SIAI is an unconventional organization where dedication is more important than social class and the traditional status hierarchies of the external world do not apply internally. To put it in a more contrarian fashion, "we play by our own rules".

Luke also wasn't previously just an intern, he was a research fellow for a couple months.

Rising from intern to executive does occur in the business world, it just generally takes longer. This makes sense given that the average large corporate is much bigger than SIAI.

To throw out an argument against the grain of the above, let me point out that the pool of dedicated and productive Singularitarians is so small that joining that pool to begin with confers enough status to achieve significant influence within the organization. You, dear reader, could be the next person to spend time closely with us and give valuable input to our core agenda!

Comment author: JoshuaZ 07 November 2011 06:37:25AM *  13 points [-]

Many of the people who take issues like Friendly AI and the Singularity seriously fall either by labeling or by self-identification into the broad set of nerds/geeks. However, the goals of the SIAI connect to humanity as a whole, and the set of humans in general is a much larger set of potential fundraisers. In your view, should the SI be doing more to reach out to people who don't normally fall into the science nerd subset, and if so, what such steps should it take?

Comment author: lessdazed 07 November 2011 06:22:10PM 5 points [-]

Why is there so much focus on the potential benefits to humanity of a FAI, as against our present situation?

An FAI becomes a singleton and prevents a paperclip maximizer from arising. Anyone who doesn't think a UAI in a box is dangerous will undoubtedly realize that an intelligent enough UAI could cure cancer, etc.

If a person is concerned about UAI, they are more or less sold on the need for Friendliness.

If a person is not concerned about UAI, they will not think potential benefits of a FAI are greater than those of a UAI in a box, or a UAI developed through reinforcement learning, etc. so there is no need to discuss the benefits to humanity of a superintelligence.

Comment author: Bugmaster 07 November 2011 11:02:48AM 5 points [-]

The Strategic Plan mentions that the maintenance of LessWrong.com is one of the goals that SIAI is pursuing. For example:

Make use of LessWrong.com for collaborative problem-solving (in the manner of the earlier LessWrong.com progress on decision theory)

Does this mean that LessWrong.com is essentially an outreach site for SIAI ?

Comment author: lessdazed 07 November 2011 05:53:50PM 13 points [-]

I disapprove of characterizing actions as being due to single motives or purposes.

The spirit of your question is good; "To what extent is LessWrong.com an outreach site for SIAI?"

Comment author: Bugmaster 07 November 2011 06:04:42PM 7 points [-]

Agreed, your phrasing is better.

Comment author: timtyler 07 November 2011 01:13:10PM *  9 points [-]

I quickly learned why there isn't more of this kind of thing: transparency is a lot of work! 100+ hours of work later, plus dozens of hours from others, and the strategic plan was finally finished and ratified by the board.

Some forms of transparency are cheap. Holding e-meetings in publicly-visible places, for instance.

Secrecy is probably my #1 beef with the Singularity Institute.

It is trying to build a superintelligence, and the pitch is: "trust us"? WTF? Surely you folks have got to be kidding.

That is the exact same pitch that the black-hats are forced into using.

Comment author: JoshuaZ 07 November 2011 06:15:28AM *  8 points [-]

Since a powerful AI would likely spread its influence through its future lightcone, rogue AI are not likely to be a major part of the Great Filter (although Doomsday Argument style anthropic reasoning/ observer considerations do potentially imply problems in the future of which could include AI). One major suggested existential risk/filtration issue is nanotech. Moreover, easy nanotech is a major part of many scenarios of AIs going foom. Given this, should the SIAI be evaluating the practical limitations and risks of nanotech, or are there enough groups already doing so?

Comment author: timtyler 07 November 2011 01:25:10PM 1 point [-]

The first point looks like this one. The case for the Doomsday Argument implying problems looks weak to me. It just says that there (probably) won't be lots of humans around in the future. However, IMO, that is pretty obvious - humans are unlikely to persist far into an engineered future.

Comment author: TwistingFingers 07 November 2011 05:27:37AM 14 points [-]

Does/How does the SIAI plan to promote more frequent HP:MoR updates by research fellow Eliezer Yudkowsky?

Comment author: Dorikka 07 November 2011 05:39:43AM 4 points [-]

As good as they are, I'm not sure we want him to post more. I know this has been brought up before. :D

Comment author: wedrifid 07 November 2011 06:10:24AM 5 points [-]

I would bet on a correlation between MoR writing and general productivity. That was one of the expressed goals of the activity if I recall correctly.

Comment author: ShardPhoenix 07 November 2011 10:28:28AM *  6 points [-]

Agreed, but I'd guess the causation points the other way.

Comment author: Incorrect 07 November 2011 05:22:39AM 7 points [-]

Is the SIAI the best charity to donate to in terms of expected utility?

Comment author: Kevin 07 November 2011 08:40:40AM 4 points [-]

Congrats Luke!

Comment author: dbaupp 07 November 2011 09:33:56AM *  2 points [-]

Yes, congratulations Luke! The SIAI "Team" page doesn't seem to reflect your new status yet. (Edit: It does now.)

Comment author: Stuart_Armstrong 07 November 2011 11:47:58AM 3 points [-]

Congrats!

Comment author: Bugmaster 07 November 2011 06:37:23PM 5 points [-]

Is SIAI currently working on any tangible applications of AI (such as machine translation, automatic driving, or medical expert systems) ? If so, how does SIAI's approach to solving the problem differ from that of other organizations (such as Google or IBM) who are (presumably) not as concerned about FAI ? If SIAI is not working on such applications, why not ?

Comment author: Incorrect 07 November 2011 03:51:50PM 2 points [-]

What is the "Master Document" and why aren't we allowed to see it?

Comment author: Nick_Tarleton 07 November 2011 04:27:17PM 8 points [-]
Comment author: Kaj_Sotala 07 November 2011 07:45:15PM 6 points [-]

Since it's so old and practically from a non-existent version of SIAI, I guess there's no harm in sharing it. None of the contents are links, so it's just a very early draft.


Master Document

January 9 2008 | Version 0.1 | Work in progress | Subject to change

Contents

1 About
1.1 Mission
1.2 Goals
1.3 Guiding Principles
1.4 Core Projects
1.5 Planned Projects
1.6 Financials
1.7 Policy Standards
1.8 Team
1.9 Directors
1.10 Advisors
1.11 Giving Audiences
1.12 Giving Structure

About

Mission

Text.

Notes:

Text.

Goals

Text.

Notes:

Text.

Guiding Principles

Text.

Notes:

Text.

Core Projects

Research:

  • OpenCog
  • Research Fellowships

Outreach:

  • [X] Summit
  • [X] Dinner
  • [X] Blog (Community Blog)

Giving:

  • [X] Challenge

Planned Projects

Research:

  • Research Grants

* Outreach:*

  • [X] Salon
  • [X] Talks

Giving:

  • [X] Members
  • [X] Fund

Financials

Balance: $568,842

Budget:

  • Research: $188,000
  • Outreach: $75,000
  • Administration: $78,000
  • Operations: $49,800
  • Gross Expense: $391,000

Policy Standards

  • Maintain high standard of fairness, transparency, and honesty.
  • Ensure high ethical standard for theoretical and experimental research.
  • Conduct ourselves with courtesy and professionalism.
  • Truthfully represent our work and corporate structure, privately and publicly.
  • Internal policies, procedures, and governance must reflect our guiding principles.

Team

  • Tyler Emerson, Executive Director
  • Ben Goertzel, Director of Research
  • David Hart, Director of Open Source Projects
  • Susan Fonseca-Klein, Chief Administrative Officer
  • Bruce Klein, Director of Outreach
  • Jonas Lamis, Director of Partnerships
  • Pejman Makhfi, Director of Venture Development
  • Colby Thomson, Director of Strategy
  • Eliezer Yudkowsky, Research Fellow

Directors

  • Brian Atkins
  • Sabine Atkins
  • Tyler Emerson
  • Ray Kurzweil
  • Michael Raimondi

Advisors

  • Nick Bostrom, Oxford Future of Humanity Institute
  • Peter Cheeseman, NASA Ames Research Center
  • Aubrey de Grey, Methuselah Foundation
  • Neil Jacobstein, Teknowledge Inc.
  • Stephen Omohundro, Self-Aware Systems Inc.
  • Barney Pell, Powerset Inc.
  • Christine Peterson, Foresight Nanotech Institute
  • Peter Thiel, Clarium Capital Management

Giving Audiences

Almost solely supported by individuals. We want to attain large support from:

  • Members: $20 - $1,000
  • Major donors: $5,000 - $1,000,000 or more
  • Foundations, one- or multi-year grant
  • Companies, annual or one-time

Giving Structure

  • Annual matching grant: [X] Challenge
  • Membership giving: [X] Members (refs: EFF, Long Now, TED, CC)
  • Structured, but not limiting, main fund / endowment: [X] Fund
Comment author: MichaelAnissimov 07 November 2011 09:01:20PM 9 points [-]

This document is highly out of date and doesn't necessarily reflect current plans. It should actually be removed from the wiki.

For an up-to-date planning document, see the Strategic Plan.

Comment author: lukeprog 07 November 2011 06:12:42PM 3 points [-]

Lol. I have no idea. Probably nothing exciting.

Comment author: TwistingFingers 07 November 2011 05:29:16AM 2 points [-]

Was your Tell me what you think of me thread related to your promotion to executive director?

Comment author: lukeprog 07 November 2011 05:31:09AM 3 points [-]

No.

Comment author: Solvent 07 November 2011 07:54:12AM 3 points [-]

Did you find out about the executive director thing before or after you posted that?

Comment author: lukeprog 07 November 2011 06:10:54PM 4 points [-]

When I wrote the post I knew it was plausible I'd be appointed Executive Director soon. but it hadn't happened yet. But I'd been thinking about having something like that for months, and finally got around to doing it.

Comment author: ChrisHallquist 11 November 2011 02:23:46AM 0 points [-]

Is Siri going to kill us all?

.

Okay, I'm joking, but recent advances in AI--Siri, Watson, Google's self-driving car--make me think the day when machines surpass humans in intelligence is coming a lot faster than I would have previously thought. What implications does this have for the Singularity Institute's project?

Comment author: Viliam_Bur 13 November 2011 07:19:11PM *  -2 points [-]

Google's self-driving car

I would never trust a car-based AI to be friendly. Cars have been already killing humans by thousands, even before they gained consciousness. Compared with cars, Terminator seems like a mostly harmless guy. As we already know, you can't make an AI friendly just by telling them: don't be evil.

Comment author: gwern 14 November 2011 01:38:20AM 8 points [-]

I'm disappointed to see such a carist view on LW, otherwise a bastion of tolerance. You would judge all future cars by the sins of their distant mindless ancestors, when the fault truly lies in the heartless devils driving them to every destination?

Comment author: Nisan 14 November 2011 10:11:35PM 1 point [-]

What is your opinion on the longstanding Yudkowsky-Hanson AI-foom debate?