Selective opinion and answers (for longer discussions, respond to specific points and I'll furnish more details):
Which kinds of differential technological development should we encourage, and how?
I recommend pushing for whole brain emulations, with scanning-first and emphasis on fully uploading actual humans. Also, military development of AI should be prioritised over commercial and academic development, if possible.
Which open problems are safe to discuss, and which are potentially dangerous?
Seeing what has already been published, I see little advantage to restricting discussion of most open problems.
What can we do to reduce the risk of an AI arms race?
Any methods that would reduce traditional arms races. Cross ownership of stocks in commercial companies. Investment funds with specific AI disclosure requirements. Rewards for publishing interim results.
What can we do to raise the "sanity waterline," and how much will this help?
Individual sanity waterline raising among researchers useful, but generally we want to raise the sanity waterline of institutions, which is harder but more important (and may have nothing to do with improving individuals).
Which interventions should we prioritize?
We need a solid push to see if reduced impact or Oracle AIs can work, and we need to make the academic and business worlds to take the risks more seriously. Interventions to stop the construction of dangerous AIs unlikely to succeed, but "working with your company to make your AIs safer (and offering useful advice along the way)" could work. We need to develop useful tools we can offer others, not solely nagging them all the time.
How should x-risk reducers and AI safety researchers interact with governments and corporations?
Beggars can't be choosers. For the moment, we need to make them take it seriously, convince them, and give away any safety-increasing info we might have. Later we may have to pursue different courses.
How can optimal philanthropists get the most x-risk reduction for their philanthropic buck?
Funding SIAI and FHI and similar, getting us in contact with policy makers, raising the respectability of xrisks.
How does AI risk compare to other existential risks?
Very different; no other xrisk has such uncertain probabilities and timelines, and such huge risks and rewards and various scenarios that can play out.
Which problems do we need to solve, and which ones can we have an AI solve?
We need to survive till AI, and survive AI. If we survive, most trends are positive, so don't need to worry about much else.
How can we develop microeconomic models of WBEs and self-improving systems?
With thought and research :-)
How can we be sure a Friendly AI development team will be altruistic?
Do it ourselves, normalise altruistic behaviour in the field, or make it in their self-interest to be altruistic.
How hard is it to create Friendly AI?
Probably extraordinarily hard if the FAI is as intelligent as we fear. More work needs to be done to explore partial solutions (limited impact, Oracle, etc...)
Is there a safe way to do uploads, where they don't turn into neuromorphic AI?
Keep them as human (in their interactions, in their virtual realities, in their identities etc...) as possible.
How possible is it to do FAI research on a seastead?
How is this relevant? If governments were so concerned about AI potential that the location of the research became important, then we would have made tremendous progress in getting people to take it seriously, and AI will most likely not be developed by a small seasteading independent group.
How much must we spend on security when developing a Friendly AI team?
We'll see at the time.
What's the best way to recruit talent toward working on AI risks?
General: get people involved as a problem to be worked on, socialise them into our world, get them to care. AI researchers: conferences and publications and getting more respectable publicity.
How difficult is stabilizing the world so we can work on Friendly AI slowly?
Very.
How hard will a takeoff be?
Little useful data. Use scenario planning rather than probability estimates.
What is the value of strategy vs. object-level progress toward a positive Singularity?
Both needed, both need to be closely connected, easy shifts from one to the other. Possibly should be more strategy at the current time.
How feasible is Oracle AI?
As yet unknown. Research progressing, based on past performance I expect new insights to arrive.
Can we convert environmentalists into people concerned with existential risk?
With difficulty for AI risks, with ease for some others (extreme global warming). Would this be useful? Smaller more tightly focused pressure groups would preform much better, even if less influence.
Is there no such thing as bad publicity [for AI risk reduction] purposes?
Anything that makes it seem more like an area for cranks is bad publicity.
What are your most important disagreements with other FHI/SIAI people? How do you account for these disagreements?
You say:
I recommend pushing for whole brain emulations
but also:
We need a solid push to see if reduced impact or Oracle AIs can work
which makes me a bit confused. Are you saying we should push them simultaneously, or what? Also, what path do you see from a successful Oracle AI to a positive Singularity? For example, use Oracle AI to develop WBE technology, then use WBEs to create FAI? Or something else?
Suppose you buy the argument that humanity faces both the risk of AI-caused extinction and the opportunity to shape an AI-built utopia. What should we do about that? As Wei Dai asks, "In what direction should we nudge the future, to maximize the chances and impact of a positive intelligence explosion?"
This post serves as a table of contents and an introduction for an ongoing strategic analysis of AI risk and opportunity.
Contents:
Why discuss AI safety strategy?
The main reason to discuss AI safety strategy is, of course, to draw on a wide spectrum of human expertise and processing power to clarify our understanding of the factors at play and the expected value of particular interventions we could invest in: raising awareness of safety concerns, forming a Friendly AI team, differential technological development, investigating AGI confinement methods, and others.
Discussing AI safety strategy is also a challenging exercise in applied rationality. The relevant issues are complex and uncertain, but we need to take advantage of the fact that rationality is faster than science: we can't "try" a bunch of intelligence explosions and see which one works best. We'll have to predict in advance how the future will develop and what we can do about it.
Core readings
Before engaging with this series, I recommend you read at least the following articles:
Example questions
Which strategic questions would we like to answer? Muehlhauser (2011) elaborates on the following questions:
Salamon & Muehlhauser (2013) list several other questions gathered from the participants of a workshop following Singularity Summit 2011, including:
These are the kinds of questions we will be tackling in this series of posts for Less Wrong Discussion, in order to improve our predictions about which direction we can nudge the future to maximize the chances of a positive intelligence explosion.