[Cross-posted at EA forum]

Hot on the heels of 80K's excellent AI risk research career profile (https://80000hours.org/career-guide/top-careers/profiles/artificial-intelligence-risk-research/), we're delighted to announce the funding of a new international Leverhulme Centre for the Future of Intelligence, to be led by Cambridge, with spokes at Oxford (Nick Bostrom), Imperial (Murray Shanahan), and Berkeley (Stuart Russell). The Centre proposal was developed by us at CSER, but will be a stand-alone centre, albeit collaborating extensively at CSER.

Building on the by-now-familiar "Puerto Rico Agenda", it will have the long-term safe and beneficial development of AI at its core, but with a slightly broader remit than CSER's focus on catastrophic AI risk and superintelligence. For example, it will consider some near-term challenges such as lethal autonomous weapons, and as well as some of the longer-term philosophical and practical issues surrounding the opportunities and challenges we expect to face, should greater-than-human-level intelligence be developed later this century.

It builds on the pioneering work of FHI, FLI and others, and the generous support of Elon Musk in massively boosting this field with his (separate) $10M grants programme in January of this year. One of the most important things this Centre will achieve is in taking a big step towards making this global area of research a long-term one in which the best talents can be expected to have lasting careers - the Centre is funded for a full 10 years, and we will aim to build longer-lasting funding on top of this.

In practical terms, it means that ~10 new postdoc positions at a minimum will be opening up in this space (we're currently pursuing matched funding opportunities) across academic disciplines and locations (Cambridge, Oxford, Berkeley, Imperial and elsewhere). Our first priority will be to identify and hire a world-class Executive Director, who would start in October. This will be a very influential position over the coming years. Research positions will most likely begin in April 2017.

In between now and then, FHI is hiring for AI safety researchers, and CSER will be hiring for an AI policy postdoc in the spring. I'll have limited time to post in between now and the Christmas break (I'll be away at NIPS and then occupied with funder deadlines and CSER recruitment), but will be happy to post more over the Christmas break if desired.

Thank you so much as always to the Lesswrong and Effective Altruism community for their support of existential risk/far future work, both financially and intellectually - it has made a huge difference over the last couple of years. Thanks in particular to MIRI and FHI's researchers, who I received a lot of guidance from in my part of co-developing this proposal.

Seán (Executive Director, CSER)

http://www.eurekalert.org/pub_releases/2015-12/uoc-cul120215.php

Human-level intelligence is familiar in biological 'hardware' -- it happens inside our skulls. Technology and science are now converging on a possible future where similar intelligence can be created in computers.

While it is hard to predict when this will happen, some researchers suggest that human-level AI will be created within this century. Freed of biological constraints, such machines might become much more intelligent than humans. What would this mean for us? Stuart Russell, a world-leading AI researcher at the University of California, Berkeley, and collaborator on the project, suggests that this would be "the biggest event in human history". Professor Stephen Hawking agrees, saying that "when it eventually does occur, it's likely to be either the best or worst thing ever to happen to humanity, so there's huge value in getting it right."

Now, thanks to an unprecedented £10 million grant from the Leverhulme Trust, the University of Cambridge is to establish a new interdisciplinary research centre, the Leverhulme Centre for the Future of Intelligence, to explore the opportunities and challenges of this potentially epoch-making technological development, both short and long term.

The Centre brings together computer scientists, philosophers, social scientists and others to examine the technical, practical and philosophical questions artificial intelligence raises for humanity in the coming century.

Huw Price, the Bertrand Russell Professor of Philosophy at Cambridge and Director of the Centre, said: "Machine intelligence will be one of the defining themes of our century, and the challenges of ensuring that we make good use of its opportunities are ones we all face together. At present, however, we have barely begun to consider its ramifications, good or bad".

The Centre is a response to the Leverhulme Trust's call for "bold, disruptive thinking, capable of creating a step-change in our understanding". The Trust awarded the grant to Cambridge for a proposal developed with the Executive Director of the University's Centre for the Study of Existential Risk (CSER), Dr Seán Ó hÉigeartaigh. CSER investigates emerging risks to humanity's future including climate change, disease, warfare and technological revolutions.

Dr Ó hÉigeartaigh said: "The Centre is intended to build on CSER's pioneering work on the risks posed by high-level AI and place those concerns in a broader context, looking at themes such as different kinds of intelligence, responsible development of technology and issues surrounding autonomous weapons and drones."

The Leverhulme Centre for the Future of Intelligence spans institutions, as well as disciplines. It is a collaboration led by the University of Cambridge with links to the Oxford Martin School at the University of Oxford, Imperial College London, and the University of California, Berkeley. It is supported by Cambridge's Centre for Research in the Arts, Social Sciences and Humanities (CRASSH). As Professor Price put it, "a proposal this ambitious, combining some of the best minds across four universities and many disciplines, could not have been achieved without CRASSH's vision and expertise."

Zoubin Ghahramani, Deputy Director, Professor of Information Engineering and a Fellow of St John's College, Cambridge, said: "The field of machine learning continues to advance at a tremendous pace, and machines can now achieve near-human abilities at many cognitive tasks -- from recognising images to translating between languages and driving cars. We need to understand where this is all leading, and ensure that research in machine intelligence continues to benefit humanity. The Leverhulme Centre for the Future of Intelligence will bring together researchers from a number of disciplines, from philosophers to social scientists, cognitive scientists and computer scientists, to help guide the future of this technology and study its implications."

The Centre aims to lead the global conversation about the opportunities and challenges to humanity that lie ahead in the future of AI. Professor Price said: "With far-sighted alumni such as Charles Babbage, Alan Turing, and Margaret Boden, Cambridge has an enviable record of leadership in this field, and I am delighted that it will be home to the new Leverhulme Centre.

[Cross-posted from EA Forum. Summary: Four new postdoc positions at the Centre for the Study of Existential Risk: Evaluation of extreme technological risk (philosophy, economics); Extreme risk and the culture of science (philosophy of science); Responsible innovation and extreme technological risk (science & technology studies, sociology, policy, governance); and an academic project manager (cutting across the Centre’s research projects, and playing a central role in Centre development). Please help us to spread the word far and wide in the academic community!]

An inspiring first recruitment round

The Centre for the Study of Existential Risk (Cambridge, UK) has been making excellent progress in building up our research team. Our previous recruitment round was a great success, and we made three exceptional hires. Dr Shahar Avin joined us in September from Google, with a background in the philosophy of science (Cambridge, UK). He is currently fleshing out several potential research projects, which will be refined and finalised following a research visit to FHI later this month. Dr Yang Liu joined us this month from Columbia University, with a background in mathematical logic and philosophical decision theory. Yang will work on problems in decision theory that relate to long-term AI, and will help us to link the excellent work being done at MIRI with relevant expertise and talent within academia. In February 2016, we will be joined by Dr Bonnie Wintle from the Centre of Excellence for Biosecurity Risk Analysis (CEBRA), who will lead our horizon-scanning work in collaboration with Professor Bill Sutherland’s group at Cambridge; among other things, she has worked on IARPA-funded development of automated horizon-scanning tools, and has been involved in the Good Judgement Project.

We are very grateful for the help of the existential risk and EA communities in spreading the word about these positions, and helping us to secure an exceptionally strong field. Additionally, I have now moved on from FHI to be CSER’s full-time Executive Director, and Huw Price is now 50% funded as CSER’s Academic Director (we share him with Cambridge’s Philosophy Faculty, where he remains Bertrand Russell Chair of Philosophy).

Four new positions:

We’re delighted to announce four new positions at the Centre for the Study of Existential Risk; details below. Unlike the previous round, where we invited project proposals from across our areas of interest, in this case we have several specific positions that we need to fill for our three year Managing Extreme Technological Risk project, funded by the Templeton World Charity Foundation; details are provided below. As we are building up our academic brand within a traditional university, we expect to predominantly hire from academia, i.e. academic researchers with (or near to the completion of) PhDs. However, we are open to hiring excellent candidates without candidates but with an equivalent and relevant level of expertise, for example in think tanks, policy settings or industry.

Three of these positions are in the standard academic postdoc mould, working on specific research projects. I’d like to draw attention to the fourth, the academic project manager. For this position, we are looking for someone with the intellectual versatility to engage across our research strands – someone who can coordinate these projects, synthesise and present our research to a range of audiences including funders, collaborators, policymakers and industry contacts. Additionally, this person will play a key role in developing the centre over the next two years, working with our postdocs and professorial advisors to secure funding, and contributing to our research, media, and policy strategy among other things. I’ve been interviewed in the past (https://80000hours.org/2013/02/bringing-it-all-together-high-impact-research-management/) about the importance of roles of this nature; right now I see it as our biggest bottleneck, and a position in which an ambitious person could make a huge difference.

We need your help – again!

In some ways, CSER has been the quietest of the existential risk organisations of late – we’ve mainly been establishing research connections, running lectures and seminars, writing research grants and building relations with policymakers (plus some behind-the scenes involvement with various projects). But we’ve been quite successful in these things, and now face an exciting but daunting level of growth: by next year we aim to have a team of 9-10 postdoctoral researchers here at Cambridge, plus senior professors and other staff. It’s very important we continue our momentum by getting world-class researchers motivated to do work of the highest impact. Reaching out and finding these people is quite a challenge, especially given our still-small team. So the help of the existential risk and EA communities in spreading the word – on your facebook feeds, on relevant mailing lists in your universities, passing them on to talented people you know – will make a huge difference to us.

Thank you so much!

Seán Ó hÉigeartaigh (Executive Director, CSER)

“The Centre for the Study of Existential Risk is delighted to announce four new postdoctoral positions for the subprojects below, to begin in January 2016 or as soon as possible afterwards. The research associates will join a growing team of researchers developing a general methodology for the management of extreme technological risk.

Evaluation of extreme technological risk will examine issues such as:

The use and limitations of approaches such as cost-benefit analysis when evaluating extreme technological risk; the importance of mitigating extreme technological risk compared to other global priorities; issues in population ethics as they relate to future generations; challenges associated with evaluating small probabilities of large payoffs; challenges associated with moral and evaluative uncertainty as they relate to the long-term future of humanity. Relevant disciplines include philosophy and economics, although suitable candidates outside these fields are welcomed. More: Evaluation of extreme technological risk

Extreme risk and the culture of science will explore the hypothesis that the culture of science is in some ways ill-adapted to successful long-term management of extreme technological risk, and investigate the option of ‘tweaking’ scientific practice, so as to improve its suitability for this special task. It will examine topics including inductive risk, use and limitations of the precautionary principle, and the case for scientific pluralism and ‘breakout thinking’ where extreme technological risk is concerned. Relevant disciplines include philosophy of science and science and technology studies, although suitable candidates outside these fields are welcomed. More: Extreme risk and the culture of science;

Responsible innovation and extreme technological risk asks what can be done to encourage risk-awareness and societal responsibility, without discouraging innovation, within the communities developing future technologies with transformative potential. What can be learned from historical examples of technology governance and culture-development? What are the roles of different forms of regulation in the development of transformative technologies with risk potential? Relevant disciplines include science and technology studies, geography, sociology, governance, philosophy of science, plus relevant technological fields (e.g., AI, biotechnology, geoengineering), although suitable candidates outside these fields are welcomed. More: Responsible innovation and extreme technological risk

We are also seeking to appoint an academic project manager, who will play a central role in developing CSER into a world-class research centre. We seek an ambitious candidate with initiative and a broad intellectual range for a postdoctoral role combining academic and administrative responsibilities. The Academic Project Manager will co-ordinate and develop CSER’s projects and the Centre’s overall profile, and build and maintain collaborations with academic centres, industry leaders and policy makers in the UK and worldwide. This is a unique opportunity to play a formative research development role in the establishment of a world-class centre. More: CSER Academic Project Manager

Candidates will normally have a PhD in a relevant field or an equivalent level of experience and accomplishment (for example, in a policy, industry, or think tank setting). Application Deadline: Midday (12:00) on November 12th 2015.”

[To be cross-posted at Effective Altruism Forum, FLI news page]

I'm delighted to announce that the Centre for the Study of Existential Risk has had considerable recent success in grantwriting and fundraising, among other activities (full update coming shortly). As a result, we are now in a position to advance to CSER's next stage of development: full research operations. Over the course of this year, we will be recruiting for a full team of postdoctoral researchers to work on a combination of general methodologies for extreme technological (and existential) risk analysis and mitigation, alongside specific technology/risk-specific projects.

Our first round of recruitment has just opened - we will be aiming to hire up to 4 postdoctoral researchers; details below. A second recruitment round will take place in the Autumn. We have a slightly unusual opportunity in that we get to cast our net reasonably wide. We have a number of planned research projects (listed below) that we hope to recruit for. However, we also have the flexibility to hire one or more postdoctoral researchers to work on additional projects relevant to CSER's aims. Information about CSER's aims and core research areas is available on our website. We request that as part of the application process potential postholders send us a research proposal of no more than 1500 words, explaining what your research skills could contribute to CSER. At this point in time, we are looking for people who will have obtained a doctorate in a relevant discipline by their start date.

We would also humbly ask that the LessWrong community aid us in spreading the word far and wide about these positions. There are many brilliant people working within the existential risk community. However, there are academic disciplines and communities that have had less exposure to existential risk as a research priority than others (due to founder effect and other factors), but where there may be people with very relevant skills and great insights. With new centres and new positions becoming available, we have a wonderful opportunity to grow the field, and to embed existential risk as a crucial consideration in all relevant fields and disciplines.

Thanks very much,

Seán Ó hÉigeartaigh (Executive Director, CSER)

"The Centre for the Study of Existential Risk (University of Cambridge, UK) is recruiting for to four full-time postdoctoral research associates to work on the project Towards a Science of Extreme Technological Risk.

We are looking for outstanding and highly-committed researchers, interested in working as part of growing research community, with research projects relevant to any aspect of the project. We invite applicants to explain their project to us, and to demonstrate their commitment to the study of extreme technological risks.

We have several shovel-ready projects for which we are looking for suitable postdoctoral researchers. These include:

• Ethics and evaluation of extreme technological risk (ETR) (with Sir Partha Dasgupta;
• Horizon-scanning and foresight for extreme technological risks (with Professor William Sutherland);
• Responsible innovation and extreme technological risk (with Dr Robert Doubleday and the Centre for Science and Policy).

However, recruitment will not necessarily be limited to these subprojects, and our main selection criterion is suitability of candidates and their proposed research projects to CSER’s broad aims.

Details are available here. Closing date: April 24th."

The Centre for the Study of Existential Risk will be holding a public lecture on "Existential Risk: Surviving the 21st Century" in collaboration with 80,000 Hours: Cambridge and Giving What We Can: Cambridge on the 26th of February in Cambridge (United Kingdom).

Lady Mitchell Hall, Sidgwick Site, Cambridge. 5:30pm-6:45pm, with drinks reception to follow.

SPEAKERS:

Lord Martin Rees, Astronomer Royal

Jaan Tallinn, co-founder of Skype

Huw Price, Bertrand Russell Professor of Philosophy at Cambridge

"In the coming century, the greatest threats to human survival may come from our own technological developments. However, if we can safely navigate the pitfalls, the benefits that technology promises are enormous. A philosopher, an astronomer, and an entrepreneur have come together to form the Centre for the Study of Existential Risk. The goal: to bring a fraction of humanity’s talents to bear on the task of ensuring our long-term survival. In this lecture, Huw Price, Martin Rees and Jaan Tallinn will outline humanity’s greatest challenge: surviving the 21st century."

This event is free and open to all.

Facebook event notice is here.

In other news, I hope to be posting a general update on progress with the Centre's establishment fairly shortly for those who are interested, although it's still an ongoing process. Things have become quite busy and there are a lot of opportunities to follow up on, so I'll be taking a leave of absence from the Future of Humanity Institute for 6 months in April to work full-time on the project to establish the centre.

The Future of Humanity Institute* recently secured funding for a new Research Collaboration with Amlin Insurance focusing on systemic risks associated with risk modelling. We're looking for someone with an academic background or interests and management/organisational abilities to coordinate and develop this project and area of research.

Who we need

This is a unique opportunity to build a world-leading research programme. We’re looking for someone who can not only manage this project, but who also has the drive and initiative to find new sources of funding, network with leading experts, and design future plans for the project. We’re also looking for someone who understands and is motivated by the aims of the FHI; the post-holder will have the opportunity to contribute across the board to FHI projects, and may be a crucial part of the FHI’s success going forward.

It’s a two year position, but there will be the possibility of extension depending on the success of the project and the acquisition of further funding. We can sponsor a visa. All the details can be found here.

Why can you make a big difference in this role?

I’ve spoken to 80,000 hours in the past about the impact a talented person can have in academic project management; this Nature article also talks about the importance of this area of work. MIRI's recent successes are also in part due to the work of some excellent people with the right mix of research understanding and organisation-running ability.

While this is not a research post, your work will increase the success and impact of research done by each one of a team of top-tier academics, and will bring yet more high-quality researchers into the most important fields. This makes this position a way to achieve a huge amount of accumulative good.  With a successful funding, development and media strategy, you can contribute to shaping the fields that the FHI is leading the world in.

More on the project

Systemic risks concern the stability of an entire market, and are of great importance to managing large-scale risk. The very methods used to model these phenomena can themselves be a source of systemic risk, especially when they embody hidden assumptions that may not remain reliable in a fast-changing world. The project will focus on gaining a better understanding of systemic risks, particularly as they apply to catastropic risk modelling, and ways to avoid or mitigate such risks. Subtopics of interest are likely to include:

• Ways in which individually rational agents can misbehave when they become part of a larger network.
• Decision making under uncertainty.
• Cognitive biases that emerge when dealing with large risks, and when using the information pipelines used to model catastrophic events.
• How to model potential existential risks.

The current project research team are: Anders Sandberg, Stuart Armstrong and Nick Beckstead.

Closing date is 19th July.

For questions about the position please email me at sean.oheigeartaigh@philosophy.ox.ac.uk; Stuart Armstrong should also be able to answer questions. I'll try to answer as many as queries as I can, but I apologise in advance if I don't get to everyone - workload at present is very heavy. We'd be very grateful for any help in spreading the word to good people who might be interested. Thank you!

*The Future of Humanity Institute

The Future of Humanity Institute (Oxford) is a world-leading research centre looking at big-picture questions for human civilization.  With the tools of mathematics, philosophy, and science, we explore the risks and opportunities that will arise from technological change, weigh ethical dilemmas, and evaluate global priorities.  Our goal is to clarify the choices that will shape humanity’s long-term future.