What are your most important disagreements with other FHI/SIAI people? How do you account for these disagreements?
Main disagreement with FHI people is that I'm more worried about AI than they are (I'm probably up with the SIAI folks on this). I suspect an anchoring effect here - I was drawn to the FHI's work through AI risk, others were drawn in through other angles (also I spend much more time on Less Wrong, making AI risks very salient). Not sure what this means for accuracy, so my considered opinion is that AI is less risky than I individually believe.
Are you saying we should push them simultaneously, or what?
My main disagreement with SIAI is that I think FAI is unlikely to be implementable on time. So I want to explore alternative avenues, several ones ideally. Oracle to FAI would be one route; Oracle to people taking AI seriously to FAI might be another. WBE opens up many other avenues (including "no AI"), so is also worth looking into.
I haven't bothered to try and close the gap between me and SIAI on this, because even if they are correct, I think it's valuable for the group to have someone looking into non-FAI avenues.
My main disagreement with SIAI is that I think FAI is unlikely to be implementable on time.
Why do you say this is a disagreement? Who at SIAI thinks FAI is likely to be implementable on time (and why)?
So I want to explore alternative avenues, several ones ideally.
Right, assuming we can find any alternative avenues of comparable probability of success. I think it's unlikely for FAI to be implementable both "on time" (i.e. by humans in current society), and via alternative avenues (of which fast WBE humans seems the most plausible one, which...
Suppose you buy the argument that humanity faces both the risk of AI-caused extinction and the opportunity to shape an AI-built utopia. What should we do about that? As Wei Dai asks, "In what direction should we nudge the future, to maximize the chances and impact of a positive intelligence explosion?"
This post serves as a table of contents and an introduction for an ongoing strategic analysis of AI risk and opportunity.
Contents:
Why discuss AI safety strategy?
The main reason to discuss AI safety strategy is, of course, to draw on a wide spectrum of human expertise and processing power to clarify our understanding of the factors at play and the expected value of particular interventions we could invest in: raising awareness of safety concerns, forming a Friendly AI team, differential technological development, investigating AGI confinement methods, and others.
Discussing AI safety strategy is also a challenging exercise in applied rationality. The relevant issues are complex and uncertain, but we need to take advantage of the fact that rationality is faster than science: we can't "try" a bunch of intelligence explosions and see which one works best. We'll have to predict in advance how the future will develop and what we can do about it.
Core readings
Before engaging with this series, I recommend you read at least the following articles:
Example questions
Which strategic questions would we like to answer? Muehlhauser (2011) elaborates on the following questions:
Salamon & Muehlhauser (2013) list several other questions gathered from the participants of a workshop following Singularity Summit 2011, including:
These are the kinds of questions we will be tackling in this series of posts for Less Wrong Discussion, in order to improve our predictions about which direction we can nudge the future to maximize the chances of a positive intelligence explosion.