I like this proposal a lot. What are the alternatives?
SI could greatly reduce public engagement, directing funds on research
As I see it, SI has two main ways to spend money: on research, and on public engagement. Obviously it has to spend money on running itself, but it's best to see that as money indirectly spent on its activities. It could direct nearly all its funding to research.
Pros: SI's research can directly bring about its central goal. We don't have all the time in the world.
Cons: Public engagement, in various ways, is what makes the research possible: it brings in the funding and it makes it easier to recruit people. In a self-sustaining non-profit, money spent on public engagement now should mean money to spend on research later. Also, public engagement directly serves the aims of SI by making more people aware of the risk.
SI could stick to presenting its existing case, leaving the gaps unfilled
Pros: That would be cheaper, and allow more to be spent on research.
Cons: given that SI's case is not intuitively appealing, making it strong seems the best way to win the right people over; as Holden Karnofsky's commentary demonstrates, leaving the holes unfilled is harming credibility and making public engagement less effective. Further, the earlier problems in this case are discovered, the more effectively future work can be directed.
SI could stick to the academic paper format, or another un-wiki "write, finish, move on" format
Pros: This presents another big cost saving: you only have to write what's new. Much of the proposed wiki content would come from work SI have already written up; there would be significant costs in adapting that work for the new format, which could be avoided if SI stick to writing new work in new papers. Furthermore, SI pretty much have to write the academic papers anyway; the work involved in writing for one format, then converting to another, can be avoided.
Cons: What you have to read to understand SI's case grows linearly. An argument made sloppily in one paper is strengthened in a later one; but you have to read both papers, notice the sloppy argument, and then reach the later paper to fix it. Or try to read the later paper, and fail to understand why this point matters, until you read the earlier one and see the context. A wiki-like "here is our whole case" format allows the case to be presented as a coherent whole, with problems with previous revisions largely elided, or relegated to specific wiki pages that need only be read by the curious.
Further, in practice the academic paper format does not free you from the need to cover old ground; in my experience finding new ways to say the same old things in the "Introduction" section of such papers introducing the problem you intend to discuss is dull and tiresome work.
I think there's lots of discussion to be had about how to get the most out of the wiki and how to minimize the costs, but as you can see, on the "is it a good idea at all" I'm pretty sold.
Series: How to Purchase AI Risk Reduction
One large project proposal currently undergoing cost-benefit analysis at the Singularity Institute is a scholarly AI risk wiki. Below I will summarize the project proposal, because:
The Idea
Think Scholarpedia:
But the scholarly AI risk wiki would differ from Scholarpedia in these respects:
Example articles: Eliezer Yudkowsky, Nick Bostrom, Ben Goertzel, Carl Shulman, Artificial General Intelligence, Decision Theory, Bayesian Decision Theory, Evidential Decision Theory, Causal Decision Theory, Timeless Decision Theory, Counterfactual Mugging, Existential Risk, Expected Utility, Expected Value, Utility, Friendly AI, Intelligence Explosion, AGI Sputnik Moment, Optimization Process, Optimization Power, Metaethics, Tool AI, Oracle AI, Unfriendly AI, Complexity of Value, Fragility of Value, Church-Turing Thesis, Nanny AI, Whole Brain Emulation, AIXI, Orthogonality Thesis, Instrumental Convergence Thesis, Biological Cognitive Enhancement, Nanotechnology, Recursive Self-Improvement, Intelligence, AI Takeoff, AI Boxing, Coherent Extrapolated Volition, Coherent Aggregated Volition, Reflective Decision Theory, Value Learning, Logical Uncertainty, Technological Development, Technological Forecasting, Emulation Argument for Human-Level AI, Evolutionary Argument for Human-Level AI, Extensibility Argument for Greater-Than-Human Intelligence, Anvil Problem, Optimality Notions, Universal Intelligence, Differential Intellectual Progress, Brain-Computer Interfaces, Malthusian Scenarios, Seed AI, Singleton, Superintelligence, Pascal's Mugging, Moore's Law, Superorganism, Infinities in Ethics, Economic Consequences of AI and Whole Brain Emulation, Creating Friendly AI, Cognitive Bias, Great Filter, Observation Selection Effects, Astronomical Waste, AI Arms Races, Normative and Moral Uncertainty, The Simulation Hypothesis, The Simulation Argument, Information Hazards, Optimal Philanthropy, Neuromorphic AI, Hazards from Large-Scale Computation, AGI Skepticism, Machine Ethics, Event Horizon Thesis, Acceleration Thesis, Singularitarianism, Subgoal Stomp, Wireheading, Ontological Crisis, Moral Divergence, Utility Indifference, Personhood Predicates, Consequentialism, Technological Revolutions, Prediction Markets, Global Catastrophic Risks, Paperclip Maximizer, Coherent Blended Volition, Fun Theory, Game Theory, The Singularity, History of AI Risk Thought, Utility Extraction, Reinforcement Learning, Machine Learning, Probability Theory, Prior Probability, Preferences, Regulation and AI Risk, Godel Machine, Lifespan Dilemma, AI Advantages, Algorithmic Complexity, Human-AGI Integration and Trade, AGI Chaining, Value Extrapolation, 5 and 10 Problem.
Most of these articles would contain previously unpublished research (not published even in blog posts or comments), because most of the AI risk research that has been done has never been written up in any form but sits in the brains and Google docs of people like Yudkowsky, Bostrom, Shulman, and Armstrong.
Benefits
More than a year ago, I argued that SI would benefit from publishing short, clear, scholarly articles on AI risk. More recently, Nick Beckstead expressed the point this way:
Chris Hallquist added:
Of course, SI has long known it could benefit from clearer presentations of its views, but the cost was too high to implement it. Scholarly authors of Nick Bostrom's skill and productivity are extremely rare, and almost none of them care about AI risk. But now, let's be clear about what a scholarly AI risk wiki could accomplish:
There are some benefits to the wiki structure in particular:
Costs
This would be a large project, and has significant costs. I'm still estimating the costs, but here are some ballpark numbers for a scholarly AI risk wiki containing all the example articles above: