Nick Bostrom takes on the facts, the fictions and the speculations in the movie Transcendence:

Could you upload Johnny Depp's brain? Oxford Professor on Transcendence

How soon until machine intelligence? Oxford professor on Transcendence

Would you have warning before artificial superintelligence? Oxford professor on Transcendence

Oxford professor on Transcendence: how could you get a machine intelligence?

For those who may be interested in these things, here are the links to all the FHI's technical reports.

Global Catastrophic Risks Survey: At the Global Catastrophic Risk Conference in Oxford (17‐20 July, 2008) an informal survey was circulated among participants, asking them to make their best guess at the chance that there will be disasters of different types before 2100. This report summarizes the main results.

Record of the Workshop on Policy Foresight and Global Catastrophic Risks: On 21 July 2008, the Policy Foresight Programme, in conjunction with the Future of Humanity Institute, hosted a day-long workshop on “Policy Foresight and Global Catastrophic Risks” at the James Martin 21st Century School at the University of Oxford. This document provides a record of the day’s discussion.

Whole Brain Emulation: a Roadmap: This report aims at providing a preliminary roadmap for Whole Brain Emulations (possible future one‐to‐one modelling of the function of the human brain), sketching out key technologies that would need to be developed or refined, and identifying key problems or uncertainties.

Utility Indifference: A utility-function-based method for making an Artificial Intelligence indifferent to certain facts or states of the world, which can be used to make certain security precautions more successful.

Machine Intelligence Survey: At the FHI Winter Intelligence conference on machine intelligence 16/1 2011 an informal poll was conducted to elicit the views of the participants on various questions related to the emergence of machine intelligence. This report summarizes the results.

Indefinite Survival through Backup Copies: Continually copying yourself may help you preserve yourself from destruction. As long as the copies fate is independent, increasing the number of copies at a logarithmic rate is enough to ensure a non-zero probability of surviving for ever. The model is of more general use for many similar processes.

Anthropics: why Probability isn’t enough: This report argues that the current treatment of anthropic and self-locating problems over-emphasises the importance of anthropic probabilities, and ignores other relevant and important factors, such as whether the various copies of the agents in question consider that they are acting in a linked fashion and whether they are mutually altruistic towards each other. These help to reinterpret the decisions, rather than probabilities, as the fundamental objects of interest in anthropic problems.

Nash equilibrium of identical agents facing the Unilateralist's Curse: This report is an addendum to the 'Unilateralist's Curse' of Nick Bostrom, Thomas Douglas and Anders Sandberg. It demonstrates that if there are identical agents facing a situation where any one of them can implement a policy unilaterally, then the best strategies they can implement are also Nash equilibriums.

AI arms race: a simple model of AI arms race (though it can be generalised). Some of the insights are obvious - that the competing teams are more likely to take safety precautions if there are not too many of them, if they agree with each other's values and if skill is more important than risk-taking in developing a functioning AI. But one result is surprising: that teams are most likely to take risks if they know the capabilities of their own team or their opponents'. In this case, the less you know, the safer you'll behave.

Please cite these reports as:

I recently gave a talk at the IARU Summer School on the Ethics of Technology.

In it, I touched on many of the research themes of the FHI: the accuracy of predictions, the limitations and biases of predictors, the huge risks that humanity may face, the huge benefits that we may gain, and the various ethical challenges that we'll face in the future.

Nothing really new for anyone who's familiar with our work, but some may enjoy perusing it.

(This is a introduction, for  those not immersed in the Singularity world, into the history of and relationships between SU, SIAI [SI, MIRI], SS, LW, CSER, FHI, and CFAR. It also has some opinions, which are strictly my own.)

The good news is that there were no Singularity Wars. 

The Bay Area had a Singularity University and a Singularity Institute, each going in a very  different direction. You'd expect to see something like the People's Front of Judea and the Judean People's Front, burning each other's grain supplies as the Romans moved in. 

We are pleased to announce a new vacancy at the Programme on the Impacts of Future Technology. Please forward to any who would be interested.

James Martin Postdoctoral Research Fellowship:

Socio-economic Impacts of Technological Change 
with the Programme on the Impacts of Future Technology

University of Oxford

Faculty of Philosophy

The Future of Humanity Institute, Oxford Martin School

Grade 7: £29,249–£39,257 per annum 
Protocol reference number: HUM/11043F/E

Applications are invited for a Research Fellowship within the Oxford Martin Programme on the Impacts of Future Technology, an interdisciplinary programme within the Oxford Martin School at Oxford University.  This Fellowship is available on a one year full-time or two years part-time fixed term basis.

The Programme, directed by Professor Nick Bostrom, analyzes possibilities related to long-range technological change and potential social impacts of future transformative technologies. Research foci include the future of computing and machine intelligence, existential risks, predictive and evaluative uncertainty, and related philosophical issues.

The postholder will conduct research on socio-economic and strategic impacts of potentially transformative or disruptive future technological innovations, including (but not limited to) advances in computing and machine intelligence, biosecurity and surveillance technology. Academic background is open. Potential areas include economics, political science, legal theory, and sociology; other relevant areas include environmental economics, game theory, and risk management. A multidisciplinary background would be favourable.

For further particulars and application details, please see:

http://www.futuretech.ox.ac.uk/vacancies

or contact:futuretech@philosophy.ox.ac.uk. 

The deadline for applications is Monday 9^th April.

After reading the FHI achievement report for 2008-2010, I thought it might be useful to compare their achievements to those of SIAI during the same time period. Since SIAI does not have an equivalent report, I've mostly pulled the data of their achievements from the SIAI blog.

My intention here is to help figure out which organization makes better use of my donations. For that purpose, I'm only looking at actual concrete outputs, and ignoring achievements such as successful fundraising drives or the hiring of extra staff.

For citation counts, I'm using Google Scholar data as-is. Note that this will include both self-cites and some cites from pages that really shouldn't be counted, since Google Scholar seems to be a bit liberal about what it includes in its database. I'm unsure as to whether or not the citation counts are very meaningful, since there hasn't been much time for anyone to cite papers published in 2010, say. But I'm including them anyway.

Future of Humanity Institute

Publications. The Achievement Report highlights three books and 22 journal articles. In addition, FHI staff has written 34 book chapters for academic volumes, including Companion to Philosophy of Technology; New Waves in Philosophy of Technology; Philosophy: Theoretical and Empirical Explorations; and Oxford Handbook of Neuroethics.

The three are the hardcover and paperback editions of Human Enhancement, as well as a paperback edition of Anthropic Bias: Observation Selection Effects in Science and Philosophy. Human Enhancement has been cited 22 times. Anthropic Bias was originally published in 2002, so I'm not including its citation count.

The highlighted 22 journal articles had been cited 59 times in total. The overwhelmingly most cited article was Cognitive Enhancement: Methods, Ethics, Regulatory Challenges in Science and Engineering Ethics, with 39 cites. The runner-up was Probing the Improbable: Methodological. Challenges for Risks with Low Probabilities and High Stake, with 5 cites. The remaining articles had 0-3 cites. But while Cognitive Enhancement is listed as a 2009 paper, it's worth noting that the first draft version of it was posted on Nick Bostrom's website back in 2006, and it has had time to accumulate cites since then. If we exclude it, FHI's 2008-2010 papers have been cited 20 times.

It's not listed in the Achievement Report, but I also want to include the 2008 Whole Brain Emulation Roadmap, which has been cited 15 times, bringing the total count (excluding Cognitive Enhancement) to 35.

Presentations. FHI members have given a total of 95 invited lectures and conference presentations.

Media appearances. Some 100 media appearances, including print, radio, and television appearances, since January 2009. These include BBC television, New Scientist, National Geographic, The Guardian, ITV, Bloomberg News, Discovery Channel, ABC, Radio Slovenia, Wired Magazine, BBC world service, Volkskrant (German newspaper), Utbildningsradion (Swedish national radio), Mehr News Agency (Iranian), Mladina Weekly (Slovenian magazine), Jyllands-Posten and Weekenavisen (Danish newspapers), Bayerisher Rundfunk (German radio), The History Channel, O Estado de São Paulo (Brazillian newspaper), Euronews, Kvallsposten (Swedish newspaper), City Helsinki (Finnish radio), Focus, Dutch Film and Television Academy, The Smart Manager (Indian magazine), Il Sole 24 Ore (Italian monthly), The Bulletin of the Atomic Sciences, Time Magazine, Astronomy Now, and Radio Bar-Kulan (Kenya).

Visitors. "The Institute receives many requests from students and scholars who wish to visit the Institute, only a few of which are accepted because of capacity limitations. The FHI has hosted a number of distinguished academic visitors over the past two years within its various areas of activity, such as Profs. David Chalmers, Michael Oppenheimer, and Thomas Homer-Dixon."

Policy advice. The Achievement Report highlights 23 groups or events which have received policy advice from either Nick Bostrom or Anders Sandberg. These include the World Economic Forum, the Public Services Offices of the Prime Minister's Office of Singapore, the UK Home Office, If (Stockholm insurance company), Jane Street Capital, IARPA (Intelligence Advanced Research Projects Activity) for US Government, The Swedish Institute for Infectious Disease Control, and setting up a research network, "A differential view of enhancement", within the Volkswagen Foundation.

Organized events. Three organized events. 1: Cognitive Enhancement Workshop. 2: Symposium on cognitive enhancement and related ethical and policy issues. 3: Uncertainty, Lags and Nonlinearity: Challenges to governance in a turbulent world.

Singularity Institute

Publications. The SIAI publications page has 15 papers from the 2008-2010 period, of which 11 are listed under "recent publications", 1 under "software", and 3 under "talks and working papers". Of these, Superintelligence does not imply benevolence has been cited once. The rest all have no citations.

The Sequences were written during this time period. They consist of about a million words, and might very well have a bigger impact than all the other FHI and SIAI articles together - though that's very hard to quantify.

Presentations and Media Appearances. The SIAI blog mentions a number of media appearances and presentations at various venues, but I don't have the energy to go through them all and count. From a quick eyeballing of the blog, though, SIAI has nowhere near as many presentations and media appearances as FHI.

Visitors. The Visiting Fellows page has a list of 27 Visiting Fellows from around the world, who attend or hold degrees from universities including Harvard, Stanford, Yale, Cambridge, Carnegie Mellon, Auckland University, Moscow Institute of Physics and Technology, and the University of California-Santa Barbara

Online communities and tools. Less Wrong was founded in 2009, and Google Analytics says that by the end of 2010, it had had over a million unique visitors.

Note that LW is an interesting case: as an FHI/SIAI collaboration, both organizations claim credit for it. However, since LW is to such a huge extent Eliezer's creation, and I'm not sure of what exactly the FHI contribution to LW is, I'm counting it as an SIAI and not a joint achievement.

SIAI also created the Uncertain Future, a tool for estimating the probability of AI.

Organized events. SIAI held Singularity Summits on all three years. The first Singularity Summit Australia was held in 2010. In 2008, SIAI co-sponsored the Converge unconference.

Ben Goertzel, acting as the SIAI Director of Research at the time, organized the 2008 and 2009 conferences on Artificial General Intelligence. He also co-organized a 2009 workshop on machine consciousness,

Artificial Intelligence projects. SIAI provided initial funding for the OpenCog project, as well as sponsoring Google Summer of Code events relating to the project in 2008 and 2009.

Overall

Based on this data, which organization is more deserving of my money? Hard to say, especially since SIAI has been changing a lot. The general AGI research, for instance, isn't really something that's being pursued anymore, and Ben Goertzel is no longer with the organization. Eliezer is no longer writing the sequences, which were possibly the biggest SIAI achievement of the whole 2008-2010 period.

Still, FHI's accomplishments seem a lot more impressive overall, suggesting that they might be a better target for the money. On the other hand, they are not as tightly focused on AI as SIAI is.

One imporant question is also the amount of funding the two organizations have had: accomplishing a lot is easier if you have more money. If an organization has thrice as much money, they should be expected to achieve thrice as much. SIAI's revenue was $426,000 in 2008 and $628,000 in 2009. FHI's funding was around $711,000 for 10/2008 - 10/2009. I don't know the 2010 figure for either organization. The FHI report also says the following:

To appreciate the significance of what has been accomplished, it should be kept in mind that the FHI has been understaffed for much of this period. One of our James Martin Research Fellows, Dr Rebecca Roache, has been on maternity leave for the past year. Our newest James Martin Research Fellow, Dr Eric Mandelbaum, who was recruited from an extremely strong field of over 170 applicants, has been in post for only two months. Thus, for half of the two-year period, FHI’s research staff has consisted of two persons, Professor Nick Bostrom and Dr Anders Sandberg.

SIAI - An Examination notes that in both 2008 and 2009, SIAI paid salaries to three people, so for a while at least, the amount of full-time staff in the two organizations was roughly comparable.

existential-risk.org

^{(Updated 2011-12-16 due to a comment by Nick Bostrom.)}

'Existential Risk FAQ' by Nick Bostrom

(2011) Version 1.0

Short answers to common questions

Link: pdf html

'Existential Risk Prevention as the Most Important Task for Humanity' by Nick Bostrom

(2011) Working paper (revised)

ABSTRACT
Existential risks are those that threaten the entire future of humanity.  Many theories of value imply that even relatively small reductions in net existential risk have enormous expected value.  Despite their importance, issues surrounding human-extinction risks and related hazards remain poorly understood.  In this paper, I clarify the concept of existential risk and develop an improved classification scheme.  I discuss the relation between existential risks and basic issues in axiology, and show how existential risk reduction (via the maxipok rule) can serve as a strongly action-guiding principle for utilitarian concerns.  I also show how the notion of existential risk suggests a new way of thinking about the ideal of sustainability.

Link: pdf html