Great list! Here are some of my beliefs;
19 How difficult is stabilizing the world so we can work on Friendly AI slowly?
Virtually impossible. The people working on AI number in the thousands, and not even governments can stop technological progress. There are probably routes to discourage AI funding, or make such work unpopular, but to talk of "stabilizing the world" is way beyond what any group of people can do.
6 What can we do to attract more funding, support, and research to x-risk reduction and to the specific sub-problems of successful Singularity navigation?
I think this is the most important question by far.
5 What can we do to raise the "sanity waterline," and how much will this help?
I really think that trying to raise the "sanity waterline", which refers to the average sanity of all people, is an enormous task which, while noble, would be a waste of time. We simply don't have the time to do it. We should focus on the people that can possibly accidentally create UFAI--academics in machine learning programs, researchers with access to supercomputers, et cetera.
If anyone disagrees, I would love to hear some evidence against what I've said. This stuff is way too complicated for me to be really confident.
I'll take you up on the disagreement. In the next 40 years, I find it very unlikely that any form of AI will be developed. Furthermore, we do want technological improvement in machine learning fields because of the advantages that can be offered in fields like self-driving cars, assisting doctors in diagnosing and treating illnesses, and many other fields.
And, because of the >40 year timeline, it will most likely be the next generation that leads the discovery to AI/FAI. So, we can't target particular people (although we can focus on those who are ...
Part of the series AI Risk and Opportunity: A Strategic Analysis.
(You can leave anonymous feedback on posts in this series here. I alone will read the comments, and may use them to improve past and forthcoming posts in this series.)
This post provides a list of questions about AI risk strategy — questions we want answered. Please suggest additional questions (a paragraph of explanation is preferred but not necessary); I may add them to the list. You can submit questions anonymously here.
Also, please identify which 3-5 of these questions you think are low-hanging fruit for productive strategic analysis on Less Wrong.
The list is in no particular order, but question numbers will remain unchanged (so that you can reliably refer to questions by their number):
What methods can we use to predict technological development? We don't yet have reliable methods for long-term technological forecasting. But not all methods have been examined yet. Perhaps technology futures have a good track record. Perhaps we could look at historical technological predictions and see if there is any pattern in the data suggesting that certain character traits and contexts lend themselves to accurate technological predictions. Perhaps there are creative solutions we haven't thought of yet.
Which kinds of differential technological development should we encourage, and how? Should we "push" on WBE, or not? Are some kinds of AI research risk-reducing, and other kinds risk-increasing? How can we achieve such effects, if they are desired?
Which open problems are safe to discuss, and which are potentially dangerous? AI risk research may itself produce risk in some cases, in the form of information hazards (Bostrom 2011). Is it safe to discuss decision theories? Acausal trade? Certain kinds of strategic questions, for example involving government intervention?
What can we do to reduce the risk of an AI arms race?
What can we do to raise the "sanity waterline," and how much will this help?
What can we do to attract more funding, support, and research to x-risk reduction and to the specific sub-problems of successful Singularity navigation?
Which interventions should we prioritize?
How should x-risk reducers and AI safety researchers interact with governments and corporations? Does Drexler's interaction with the U.S. government regarding molecular nanotechnology provide any lessons for how AI risk researchers should act?
How can optimal philanthropists get the most x-risk reduction for their philanthropic buck?
How does AI risk compare to other existential risks?
Which problems do we need to solve, and which ones can we have an AI solve?
How can we develop microeconomic models of WBEs and self-improving systems?
How can we be sure a Friendly AI development team will be altruistic?
How hard is it to create Friendly AI?
What is the strength of feedback from neuroscience to AI rather than brain emulation?
Is there a safe way to do uploads, where they don't turn into neuromorphic AI?
How much must we spend on security when developing a Friendly AI team?
What's the best way to recruit talent toward working on AI risks?
How difficult is stabilizing the world so we can work on Friendly AI slowly?
How hard will a takeoff be? To what degree is "intelligence" (as efficient cross-domain optimization) a matter of content vs. algorithms? How much does takeoff depend on slow, real-world experiments?
What is the value of strategy vs. object-level progress toward a positive Singularity?
What different kinds of Oracle AI are there, and are any of them both safe and feasible?
How much should we be worried about "metacomputational hazards"? E.g. should we worry about nonperson predicates? Oracle AIs engaging in self-fulfilling prophecies? Acausal hijacking?
What improvements can we make to the way we go about answering strategy questions? Wei Dai's notes on this question: "For example, should we differentiate between "strategic insights" (such as Carl Shulman's insight that WBE-based Singletons may be feasible) and "keeping track of the big picture" (forming the overall strategy and updating it based on new insights and evidence), and aim to have people specialize in each, so that people deciding strategy won't be tempted to overweigh their own insights? Another example: is there a better way to combine probability estimates from multiple people?"
How do people in other fields answer strategy questions? Wei Dai's notes on this question: "Is there such a thing as a science or art of strategy that we can copy from (and perhaps improve upon with ideas from x-rationality)?"
[more questions to come, as they are posted to the comments section]