(This is a introduction, for  those not immersed in the Singularity world, into the history of and relationships between SU, SIAI [SI, MIRI], SS, LW, CSER, FHI, and CFAR. It also has some opinions, which are strictly my own.)

The good news is that there were no Singularity Wars. 

The Bay Area had a Singularity University and a Singularity Institute, each going in a very  different direction. You'd expect to see something like the People's Front of Judea and the Judean People's Front, burning each other's grain supplies as the Romans moved in. 

Back in May, Luke suggested the creation of a scholarly AI risk wiki, which was to include a large set of summary articles on topics related to AI risk, mapped out in terms of how they related to the central debates about AI risk. In response, Wei Dai suggested that among other things, the existing Less Wrong wiki could be improved instead. As a result, the Singularity Institute has massively improved the LW wiki, in preparation for a more ambitious scholarly AI risk wiki. The outcome was the creation or dramatic expansion of the following articles:

• 5-and-10
• Acausal Trade
• Acceleration thesis
• Agent
• AGI chaining
• AGI skepticism
• AGI Sputnik moment
• AI advantages
• AI arms race
• AI Boxing
• AI-complete
• AI takeoff
• AIXI
• Algorithmic complexity
• Anvil problem
• Astronomical waste
• Bayesian decision theory
• Benevolence
• Ben Goertzel
• Bias
• Biological Cognitive Enhancement
• Brain-computer interfaces
• Carl Shulman
• Causal decision theory
• Church-Turing thesis
• Coherent Aggregated Volition
• Coherent Blended Volition
• Coherent Extrapolated Volition
• Computing overhang
• Computronium
• Consequentialism
• Counterfactual mugging
• Creating Friendly AI
• Cyc
• Decision theory
• Differential intellectual progress
• Economic consequences of AI and whole brain emulation
• Eliezer Yudkowsky
• Empathic inference
• Emulation argument for human-level AI
• EURISKO
• Event horizon thesis
• Evidential Decision Theory
• Evolutionary algorithm
• Evolutionary argument for human-level AI
• Existential risk
• Expected utility
• Expected value
• Extensibility argument for greater-than-human intelligence
• FAI-complete
• Fallacy
• Fragility_of_value
• Friendly AI
• Fun Theory
• Future of Humanity Institute
• Game theory
• Gödel machine
• Great Filter
• History of AI risk thought
• Human-AGI integration and trade
• Induction
• Infinities in ethics
• Information hazard
• Instrumental convergence thesis
• Intelligence
• Intelligence explosion
• Jeeves Problem
• Lifespan dilemma
• Machine ethics
• Machine learning
• Malthusian Scenarios
• Metaethics
• Moore's law
• Moral divergence
• Moral uncertainty
• Nanny AI
• Nanotechnology
• Neuromorphic AI
• Nick Bostrom
• Nonperson predicate
• Observation selection effect
• Ontological crisis
• Optimal philanthropy
• Optimization power
• Optimization process
• Oracle AI
• Orthogonality thesis
• Paperclip maximizer
• Pascal's mugging
• Prediction market
• Preference
• Prior probability
• Probability theory
• Recursive self-improvement
• Reflective decision theory
• Regulation and AI risk
• Reinforcement learning
• Search space
• Seed AI
• Self Indication Assumption
• Self Sampling Assumption
• Scoring rule
• Simulation argument
• Simulation hypothesis
• Singleton
• Singularitarianism
• Singularity
• Subgoal stomp
• Superintelligence
• Superorganism
• Technological forecasting
• Technological revolution
• Terminal value
• Timeless decision theory
• Tool AI
• Unfriendly AI
• Universal intelligence
• Utility
• Utility extraction
• Utility indifference
• Value extrapolation
• Value learning
• Whole brain emulation
• Wireheading

In managing the project, I focused on content over presentation, so a number of articles still have minor issues such as the grammar and style having room for improvement. It's our hope that, with the largest part of the work already done, the LW community will help improve the articles even further.

Thanks to everyone who worked on these pages: Alex Altair, Adam Bales, Caleb Bell, Costanza Riccioli, Daniel Trenor, João Lourenço, Joshua Fox, Patrick Rhodes, Pedro Chaves, Stuart Armstrong, and Steven Kaas.

Progress updates are nice, but without a previously defined metric for success it's hard to know whether an organization's achievements are noteworthy or not. Is SI making good progress, or underwhelming progress?

Luckily, in August 2011 we published a strategic plan that outlined lots of specific goals. It's now almost August 2012, so we can check our progress against the standard set nearly one year ago. The plan doesn't specify a timeline for the stated goals, but I remember hoping that we could do most of them by the end of 2012, while understanding that we should list more goals than we could actually accomplish given current resources.

Let's walk through the goals in that strategic plan, one by one. (Or, you can skip to the "summary and path forward" section at the end.)

1.1. Clarify the open problems relevant to our core mission

This was accomplished to some degree with So You Want to Save the World, and is on track to be accomplished to a greater degree with Eliezer's sequence "Open Problems in Friendly AI," which you should begin seeing late in August.

1.2. Identify and recruit researcher candidates who can solve research problems.

Several strategies for doing this were listed, but the only one worth doing at our current level of funding was to recruit more research associates and hire more researchers. Since August 2011 we have done both, adding half a dozen research associates and hiring nearly a dozen remote researchers, including a few who are working full-time on papers and other projects (e.g. Kaj Sotala).

1.3. Use researchers and research associates to solve open problems related to Friendly AI theory.

I never planned to be doing this by the end of 2012; it's more of a long-term goal. A first step in this direction is to have Eliezer transition back to FAI work, e.g. with his "Open Problems in Friendly AI" Summit 2011 talk and forthcoming blog sequence. And actually, SI research associate Vladimir Slepnev has been making interesting progress in LW-style decision theory, and is working on a paper explicating one of his results. (Some credit is due to Vladimir Nesov and others.)

1.4. Estimate current AI risk levels.

Alas, we haven't done much of this. There's some analysis in Intelligence Explosion: Evidence and Import, Reply to Holden on Tool AI, and Reply to Holden on The Singularity Institute. Also, Anna is working on a simple model of AI risk in MATLAB (or some similar program). But I would have liked to have the cash to hire a researcher to continue things like AI Risk and Opportunity: A Strategic Analysis.

2.1. Continue operation of the Singularity Summit, which is beginning to yield a profit while also reaching more people with our message.

We did run Singularity Summit 2011, and Singularity Summit 2012 is on track to be noticeably more fun and professional than all past Summits. (So, register now!)

The strategic plan listed subgoals of gaining corporate sponsors and possibly expanding the Summit outside the USA. We gained corporate sponsors for Summit 2011, and are on track to gain even more of them for Summit 2012. Early in 2011 we also pursued an opportunity to host the first Singularity Summit in Europe, but the financing didn't quite come through.

2.2 Cultivate LessWrong.com and the greater rationality community as a resource for Singularity Institute.

The strategic plan lists 5 subgoals, all of which we achieved. SI (a) used LessWrong to recruit additional supporters, (b) made use of LessWrong for collaborative problem solving (e.g. this and this), (c) published lots of top-level posts, (d) and published How to Run a Successful Less Wrong Meetup Group. The early efforts of CFAR, and our presence at (e.g.) Skepticon IV, made headway on 2.2.e: "Encourage improvements in critical thinking in the wider world. We need a larger community of critical thinkers for use in recruiting, project implementation, and fundraising."

2.3. Spread our message and clarify our arguments with public-facing academic deliverables.

We did exceptionally well on this, though much more is needed. In addition to detailed posts like Reply to Holden on Tool AI and Reply to Holden on the Singularity Institute, SI has more peer-reviewed publications in 2012 than in all past years combined.

2.4. Build more relationships with the optimal philanthropy, humanist, and critical thinking communities, which share many of our values.

Though this work has been mostly invisible, Carl Shulman has spent dozens of hours on building relationships with the optimal philanthropy community. We've also built relationships with the humanist and critical thinking communities, through our presence at Skepticon IV but especially through the early activities of CFAR.

2.5. Cultivate and expand Singularity Institute’s Volunteer Program.

SI's volunteer program got a new website (though we'd like to launch another redesign soon), and we estimate that SI volunteers have done 2x-5x more work per month this year than in the past few years.

2.6. Improve Singularity Institute’s web presence.

Done. We got a new domain, Singularity.org, and put up a new website there. We produced additional introductory materials, like Friendly-AI.com and IntelligenceExplosion.com. We produced lots of "landing pages," for example our tech summaries. We did not, however, complete subgoals (d) and (e) — "Continue to produce articles on targeted websites and other venues" and "Produce high-quality videos to explain Singularity Institute’s mission" — because their ROI isn't high enough to do at our current funding level.

2.7. Apply for grants, especially ones that are given to other organizations and researchers concerned with the safety of future technologies (e.g. synthetic biology and nanotechnology).

This one was always meant as a longer-range goal. SI still needs to be "fixed up" in certain ways before this is worth trying.

2.8. Continue targeted interactions with the public.

We didn't do much of this, either. In particular, Eliezer's rationality books are on hold for now; we have the author of a best-selling science book on retainer to take a crack at Eliezer's rationality books this fall, after he completes his current project.

2.9. Improve interactions with current and past donors.

Success. We created and cleaned up our donor database, communicated more regularly with our support base (previously via monthly updates and now our shiny new newsletter, which you can sign up for here), and updated our top donors list.

3.1. Encourage a new organization to begin rationality instruction similar to what Singularity Institute did in 2011 with Rationality Minicamp and Rationality Boot Camp.

This is perhaps the single most impressive thing we did this year, in the sense that it required dozens of smaller pieces to all work, and work together. The organization is now called the Center for Applied Rationality (CFAR), and it was recently approved for 501c3 status. It has its own website, has been running extremely well-reviewed rationality retreats, and has lots more exciting stuff going on that hasn't been described online yet. Sign up for CFAR's newsletter to get these juicy details when they are written up.

3.2. Use Charity Navigator’s guidelines to improve financial and organizational transparency and efficiency.

There are 9 subgoals listed here. We've since decided we don't want to grow to five independent board members (subgoal b) at this time, because a smaller board runs more efficiently. (I've now heard too many nightmare stories about trying to get things done with a large board.) We did achieve (a), (d), (e), (g), (h), and (i). Subgoal (c) is a longer term goal that we are working toward (we need a professional bookkeeper to clean up our internal processes before we can have a hired CPA audit, and we're interviewing bookkeepers now). Subgoal (f) — a records retention policy — is in the works.

3.3. Ensure a proper orientation for new Singularity Institute staff and visiting fellows.

This is in process; we're creating orientation materials.

3.4. Secure lines of credit to increase liquidity and smooth out the recurring cash-flow pinches that result from having to do things like make payroll and rent event spaces.

We've done this.

3.5. Improve safe return on financial reserves

For starters, we put a large chunk of our resources in an ING Direct high-interest savings account.

3.6. Ensure high standards for staff effectiveness.

There are two subgoals here. Subgoal (b) was to have staff maintain work logs, which we've been doing for many months now. Subgoal (a) is more ambiguous. We haven't given people job descriptions because at such a small organization, such roles change quickly. But I do provide stronger management of SI staff and projects than ever before, and this clarifies the expectations for our staff, often including task and project deadlines.

3.7. When hiring, advertise for applications to find the best candidates.

We've been doing this for several months now, e.g. here and here.

Summary

That's it for the main list! Now let's check in on what we said our top priorities for 2011-2012 were:

1. Public-facing research on creating a positive singularity. Check. SI has more peer-reviewed publications in 2012 than in all past years combined.
2. Outreach / education / fundraising. Check. Especially, through CFAR.
3. Improved organizational effectiveness. Check. Lots of good progress on this.
4. Singularity Summit. Check.

In summary, I think SI is a bit behind where I hoped we'd be by now, though this is largely because we've poured so much into launching CFAR, and as a result, CFAR has turned out to be significantly more cool at launch than I had anticipated.

Fundraising has been a challenge. One donor failed to actually give their $46,000 pledge despite repeated reminders and requests, and our support base is (understandably) anxious to see a shift from movement-building work to FAI research, a shift I have been fighting for since I was made Executive Director. (Note that spinning off rationality work to CFAR is a substantial part of trimming SI down into being primarily an FAI research institute.)

Reforming SI into a more efficient, effective organization has been my greatest challenge. Frankly, SI was in pretty bad shape when Louie and I arrived as interns in April 2011, and there have been an incredible number of holes to dig SI out of — and several more remain. (In contrast, it has been a joy to help set up CFAR properly from the very beginning, with all the right organizational tools and processes in place.) Reforming SI presents a fundraising problem, because reforming SI is time consuming and sometimes costly, but is generally unexciting to donors. I can see the light at the end of the tunnel, though. We won't reach it if we can't improve our fundraising success in the next 3-6 months, but it's close enough that I can see it.

SI's path forward, from my point of view, looks like this:

1. We finish launching CFAR, which takes over the rationality work SI was doing. (Before January 2013.)
2. We change how the Singularity Summit is planned and run so that it pulls our core staff away from core mission work to a lesser degree. (Before January 2013.)
3. Eliezer writes the "Open Problems in Friendly AI" sequence. (Before January 2013.)
4. We hire 1-2 researchers to produce technical write-ups from Eliezer's TDT article and from his "Open Problems in Friendly AI" sequence. (Beginning September 2012, except that right now we don't have the cash to hire the 1-2 people who I know who could do this and who want to do this as soon as we have the money to hire them.)
5. With the "Open FAI Problems" sequence and the technical write-ups in hand, we greatly expand our efforts to show math/compsci researchers that there is a tractable, technical research program in FAI theory, and as a result some researchers work on the sexiest of these problems from their departments, and some other math researchers take more seriously the prospect of being hired by SI to do technical research in FAI theory. (Beginning, roughly, in April 2013.) Also: There won't be classes on x-risk at SPARC (rationality camp for young elite math talent), but some SPARC students might end up being interested in FAI stuff by osmosis. 
6. With a more tightly honed SI, improved fundraising practices, and visible mission-central research happening, SI is able to attract more funding and hire even more FAI researchers. (Beginning, roughly, in September 2013.)

If you want to help us make this happen, please donate during our July matching drive!

Series: How to Purchase AI Risk Reduction

One large project proposal currently undergoing cost-benefit analysis at the Singularity Institute is a scholarly AI risk wiki. Below I will summarize the project proposal, because:

• I would like feedback from the community on it, and
• I would like to provide just one example of the kind of x-risk reduction that can be purchased with donations to the Singularity Institute.

The Idea

Think Scholarpedia:

• Open-access scholarly articles written at roughly the "Scientific American" level of difficulty.
• Runs on MediaWiki, but articles can only be created and edited by carefully selected authors, and curated by experts in the domain relevant to each article. (The editors would be SI researchers at first, and most of the authors and contributors would be staff researchers, research associates, or "remote researchers" from SI.)

But the scholarly AI risk wiki would differ from Scholarpedia in these respects:

• Is focused on the subject of AI risk and related subjects.
• No formal peer review system. The articles would, however, be continuously revised in response to comments from experts in the relevant fields, many of whom already work in the x-risk field or are knowledgeable participants on LessWrong and in the SIAI/FHI/etc. communities.
• Articles will be written for a broader educated audience, not just for domain experts. (Many articles on Scholarpedia aren't actually written at the Scientific American level, despite that stated intent.)
• A built-in citations and references system, Biblio (perhaps with the BibTeX addition).

Example articles: Eliezer Yudkowsky, Nick Bostrom, Ben Goertzel, Carl Shulman, Artificial General Intelligence, Decision Theory, Bayesian Decision Theory, Evidential Decision Theory, Causal Decision Theory, Timeless Decision Theory, Counterfactual Mugging, Existential Risk, Expected Utility, Expected Value, Utility, Friendly AI, Intelligence Explosion, AGI Sputnik Moment, Optimization Process, Optimization Power, Metaethics, Tool AI, Oracle AI, Unfriendly AI, Complexity of Value, Fragility of Value, Church-Turing Thesis, Nanny AI, Whole Brain Emulation, AIXI, Orthogonality Thesis, Instrumental Convergence Thesis, Biological Cognitive Enhancement, Nanotechnology, Recursive Self-Improvement, Intelligence, AI Takeoff, AI Boxing, Coherent Extrapolated Volition, Coherent Aggregated Volition, Reflective Decision Theory, Value Learning, Logical Uncertainty, Technological Development, Technological Forecasting, Emulation Argument for Human-Level AI, Evolutionary Argument for Human-Level AI, Extensibility Argument for Greater-Than-Human Intelligence, Anvil Problem, Optimality Notions, Universal Intelligence, Differential Intellectual Progress, Brain-Computer Interfaces, Malthusian Scenarios, Seed AI, Singleton, Superintelligence, Pascal's Mugging, Moore's Law, Superorganism, Infinities in Ethics, Economic Consequences of AI and Whole Brain Emulation, Creating Friendly AI, Cognitive Bias, Great Filter, Observation Selection Effects, Astronomical Waste, AI Arms Races, Normative and Moral Uncertainty, The Simulation Hypothesis, The Simulation Argument, Information Hazards, Optimal Philanthropy, Neuromorphic AI, Hazards from Large-Scale Computation, AGI Skepticism, Machine Ethics, Event Horizon Thesis, Acceleration Thesis, Singularitarianism, Subgoal Stomp, Wireheading, Ontological Crisis, Moral Divergence, Utility Indifference, Personhood Predicates, Consequentialism, Technological Revolutions, Prediction Markets, Global Catastrophic Risks, Paperclip Maximizer, Coherent Blended Volition, Fun Theory, Game Theory, The Singularity, History of AI Risk Thought, Utility Extraction, Reinforcement Learning, Machine Learning, Probability Theory, Prior Probability, Preferences, Regulation and AI Risk, Godel Machine, Lifespan Dilemma, AI Advantages, Algorithmic Complexity, Human-AGI Integration and Trade, AGI Chaining, Value Extrapolation, 5 and 10 Problem.

Most of these articles would contain previously unpublished research (not published even in blog posts or comments), because most of the AI risk research that has been done has never been written up in any form but sits in the brains and Google docs of people like Yudkowsky, Bostrom, Shulman, and Armstrong.

Benefits

More than a year ago, I argued that SI would benefit from publishing short, clear, scholarly articles on AI risk. More recently, Nick Beckstead expressed the point this way:

Most extant presentations of SIAI's views leave much to be desired in terms of clarity, completeness, concision, accessibility, and credibility signals.

Chris Hallquist added:

I've been trying to write something about Eliezer's debate with Robin Hanson, but the problem I keep running up against is that Eliezer's points are not clearly articulated at all. Even making my best educated guesses about what's supposed to go in the gaps in his arguments, I still ended up with very little.

Of course, SI has long known it could benefit from clearer presentations of its views, but the cost was too high to implement it. Scholarly authors of Nick Bostrom's skill and productivity are extremely rare, and almost none of them care about AI risk. But now, let's be clear about what a scholarly AI risk wiki could accomplish:

• Provide a clearer argument for caring about AI risk. Journal-published articles like Chalmers (2010) can be clear and scholarly, but the linear format is not ideal for analyzing such a complex thing as AI risk. Even a 65-page article like Chalmers (2010) can't hope to address even the tiniest fraction of the relevant evidence and arguments. Nor can it hope to respond to the tiniest fraction of all the objections that are "obvious" to some of its readers. What we need is a modular presentation of the evidence and the arguments, so that those who accept physicalism, near-term AI, and the orthogonality thesis can jump right to the sections on why various AI boxing methods may not work, while those who aren't sure what to think of AI timelines can jump to those articles, and those who accept most of the concern for AI risk but think there's no reason to assert humane values over arbitrary machine values can jump to the article on that subject. (Note that I don't presume all the analysis that would go into building an AI risk wiki would end up clearly recommending SI's current, very specific positions on AI risk, but I'm pretty sure it would clearly recommend some considerable concern for AI risk.)
• Provide a clearer picture of our AI risk situation. Without clear presentations of most of the relevant factors, it is very costly for interested parties to develop a clear picture of our AI risk situation. If you wanted to get roughly as clear a picture of our AI risk situation as can be had today, you'd have to (1) read several books, hundreds of articles and blog posts, and the archives of SI's decision theory mailing list and several forums, (2) analyze them in detail to try to fill in all the missing steps in the reasoning presented in these sources, and (3) have dozens of hours of conversation with the leading experts in the field (Yudkowsky, Bostrom, Shulman, Armstrong, etc.). With a scholarly AI risk wiki, a decently clear picture of our AI risk situation will be much cheaper to acquire. Indeed, it will almost certainly clarify the picture of our situation even for the leading experts in the field.
• Make it easier to do AI risk research. A researcher hoping to do AI risk research is in much the same position as the interested reader hoping to gain a clearer picture of our AI risk situation. Most of the relevant material is scattered across hundreds of books, articles, blog posts, forum comments, mailing list messages, and personal conversations. And those presentations of the ideas leave "much to be desired in terms of clarity, completeness, concision, accessibility..." This makes it hard to do research, in big-picture conceptual ways, but also in small, annoying ways. What paper can you cite on Thing X and Thing Y? When the extant scholarly literature base is small, you can't cite the sources that other people have dug up already. You have to do all that digging yourself.

There are some benefits to the wiki structure in particular:

• Some wiki articles can largely be ripped/paraphrased from existing papers like Chalmers (2010) and Muehlhauser & Salamon (2012).
• Many wiki articles can be adapted to become journal articles, if they are seen as having much value. Probably, 1-3 wiki articles could be developed, then adapted and combined into a journal article and published, and then the original wiki article(s) could be published on the wiki (while citing the now-published journal article).
• It's not an all-or-nothing project. Some value is gained by having some articles on the wiki, more value is gained by having more articles on the wiki.
• There are robust programs and plugins for managing this kind of project (MediaWiki, Biblio, etc.)
• Dozens or hundreds of people can contribute, though they will all be selected by editors. (SI's army of part-time remote researchers is already more than a dozen strong, each with different skills and areas of domain expertise.)

Costs

This would be a large project, and has significant costs. I'm still estimating the costs, but here are some ballpark numbers for a scholarly AI risk wiki containing all the example articles above:

• 1,920 hours of SI staff time (80 hrs/week for 24 months). This comes out to about $48,000, depending on who is putting in these hours.
• $384,000 paid to remote researchers and writers ($16,000/mo for 24 months; our remote researchers generally work part-time, and are relatively inexpensive).
• $30,000 for wiki design, development, hosting costs

(The following is a summary of some of my previous submissions that I originally created for my personal blog.)

As we know,
There are known knowns.
There are things
We know we know.
We also know
There are known unknowns.
That is to say
We know there are some things
We do not know.
But there are also unknown unknowns,
The ones we don’t know
We don’t know.

— Donald Rumsfeld, Feb. 12, 2002, Department of Defense news briefing

Intelligence, a cornucopia?

It seems to me that those who believe into the possibility of catastrophic risks from artificial intelligence act on the unquestioned assumption that intelligence is kind of a black box, a cornucopia that can sprout an abundance of novelty. But this implicitly assumes that if you increase intelligence you also decrease the distance between discoveries.

Intelligence is no solution in itself, it is merely an effective searchlight for unknown unknowns and who knows that the brightness of the light increases proportionally with the distance between unknown unknowns? To enable an intelligence explosion the light would have to reach out much farther with each increase in intelligence than the increase of the distance between unknown unknowns. I just don’t see that to be a reasonable assumption.

Intelligence amplification, is it worth it?

It seems that if you increase intelligence you also increase the computational cost of its further improvement and the distance to the discovery of some unknown unknown that could enable another quantum leap. It seems that you need to apply a lot more energy to get a bit more complexity.

If any increase in intelligence is vastly outweighed by its computational cost and the expenditure of time needed to discover it then it might not be instrumental for a perfectly rational agent (such as an artificial general intelligence), as imagined by game theorists, to increase its intelligence as opposed to using its existing intelligence to pursue its terminal goals directly or to invest its given resources to acquire other means of self-improvement, e.g. more efficient sensors.

What evidence do we have that the payoff of intelligent, goal-oriented experimentation yields enormous advantages (enough to enable an intelligence explosion) over evolutionary discovery relative to its cost?

We simply don’t know if intelligence is instrumental or quickly hits diminishing returns.

Can intelligence be effectively applied to itself at all? How do we know that any given level of intelligence is capable of handling its own complexity efficiently? Many humans are not even capable of handling the complexity of the brain of a worm.

Humans and the importance of discovery

There is a significant difference between intelligence and evolution if you apply intelligence to the improvement of evolutionary designs:

• Intelligence is goal-oriented.
• Intelligence can think ahead.
• Intelligence can jump fitness gaps.
• Intelligence can engage in direct experimentation.
• Intelligence can observe and incorporate solutions of other optimizing agents.

But when it comes to unknown unknowns, what difference is there between intelligence and evolution? The critical similarity is that both rely on dumb luck when it comes to genuine novelty. And where else but when it comes to the dramatic improvement of intelligence itself does it take the discovery of novel unknown unknowns?

We have no idea about the nature of discovery and its importance when it comes to what is necessary to reach a level of intelligence above our own, by ourselves. How much of what we know was actually the result of people thinking quantitatively and attending to scope, probability, and marginal impacts? How much of what we know today is the result of dumb luck versus goal-oriented, intelligent problem solving?

Our “irrationality” and the patchwork-architecture of the human brain might constitute an actual feature. The noisiness and patchwork architecture of the human brain might play a significant role in the discovery of unknown unknowns because it allows us to become distracted, to leave the path of evidence based exploration.

A lot of discoveries were made by people who were not explicitly trying to maximizing expected utility. A lot of progress is due to luck, in the form of the discovery of unknown unknowns.

A basic argument in support of risks from superhuman intelligence is that we don’t know what it could possible come up with. That is also why it is called it a “Singularity“. But why does nobody ask how a superhuman intelligence knows what it could possible come up with?

It is not intelligence in and of itself that allows humans to accomplish great feats. Even people like Einstein, geniuses who were apparently able to come up with great insights on their own, were simply lucky to be born into the right circumstances, the time was ripe for great discoveries, thanks to previous discoveries of unknown unknowns.

Evolution versus Intelligence

It is argued that the mind-design space must be large if evolution could stumble upon general intelligence and that there are low-hanging fruits that are much more efficient at general intelligence than humans are, evolution simply went with the first that came along. It is further argued that evolution is not limitlessly creative, each step must increase the fitness of its host, and that therefore there are artificial mind designs that can do what no product of natural selection could accomplish.

I agree with the above, yet given all of the apparent disadvantages of the blind idiot God, evolution was able to come up with altruism, something that works two levels above the individual and one level above society. So far we haven’t been able to show such ingenuity by incorporating successes that are not evident from an individual or even societal position.

The example of altruism provides evidence that intelligence isn’t many levels above evolution. Therefore the crucial question is, how great is the performance advantage? Is it large enough to justify the conclusion that the probability of an intelligence explosion is easily larger than 1%? I don’t think so. To answer this definitively we would have to fathom the significance of the discovery (“random mutations”) of unknown unknowns in the dramatic amplification of intelligence versus the invention (goal-oriented “research and development”) of an improvement within known conceptual bounds.

Another example is flight. Artificial flight is not even close to the energy efficiency and maneuverability of birds or insects. We didn’t went straight from no artificial flight towards flight that is generally superior to the natural flight that is an effect of biological evolution.

Take for example a dragonfly. Even if we were handed the design for a perfect artificial dragonfly, minus the design for the flight of a dragonfly, we wouldn’t be able to build a dragonfly that can take over the world of dragonflies, all else equal, by means of superior flight characteristics.

It is true that a Harpy Eagle can lift more than three-quarters of its body weight while the Boeing 747 Large Cargo Freighter has a maximum take-off weight of almost double its operating empty weight (I suspect that insects can do better). My whole point is that we never reached artificial flight that is strongly above the level of natural flight. An eagle can after all catch its cargo under various circumstances like the slope of a mountain or from beneath the sea, thanks to its superior maneuverability.

Humans are biased and irrational

It is obviously true that our expert systems are better than we are at their narrow range of expertise. But that expert systems are better at certain tasks does not imply that you can effectively and efficiently combine them into a coherent agency.

The noisiness of the human brain might be one of the important features that allows it to exhibit general intelligence. Yet the same noise might be the reason that each task a human can accomplish is not put into execution with maximal efficiency. An expert system that features a single stand-alone ability is able to reach the unique equilibrium for that ability. Whereas systems that have not fully relaxed to equilibrium feature the necessary characteristics that are required to exhibit general intelligence. In this sense a decrease in efficiency is a side-effect of general intelligence. If you externalize a certain ability into a coherent framework of agency, you decrease its efficiency dramatically. That is the difference between a tool and the ability of the agent that uses the tool.

In the above sense, our tendency to be biased and act irrationally might partly be a trade off between plasticity, efficiency and the necessity of goal-stability.

Embodied cognition and the environment

Another problem is that general intelligence is largely a result of an interaction between an agent and its environment. It might be in principle possible to arrive at various capabilities by means of induction, but it is only a theoretical possibility given unlimited computational resources. To achieve real world efficiency you need to rely on slow environmental feedback and make decision under uncertainty.

AIXI is often quoted as a proof of concept that it is possible for a simple algorithm to improve itself to such an extent that it could in principle reach superhuman intelligence. AIXI proves that there is a general theory of intelligence. But there is a minor problem, AIXI is as far from real world human-level general intelligence as an abstract notion of a Turing machine with an infinite tape is from a supercomputer with the computational capacity of the human brain. An abstract notion of intelligence doesn’t get you anywhere in terms of real-world general intelligence. Just as you won’t be able to upload yourself to a non-biological substrate because you showed that in some abstract sense you can simulate every physical process.

Just imagine you emulated a grown up human mind and it wanted to become a pick up artist, how would it do that with an Internet connection? It would need some sort of avatar, at least, and then wait for the environment to provide a lot of feedback.

Therefore even if we’re talking about the emulation of a grown up mind, it will be really hard to acquire some capabilities. Then how is the emulation of a human toddler going to acquire those skills? Even worse, how is some sort of abstract AGI going to do it that misses all of the hard coded capabilities of a human toddler?

Can we even attempt to imagine what is wrong about a boxed emulation of a human toddler, that makes it unable to become a master of social engineering in a very short time?

Can we imagine what is missing that would enable one of the existing expert systems to quickly evolve vastly superhuman capabilities in its narrow area of expertise? Why haven’t we seen a learning algorithm teaching itself chess intelligence starting with nothing but the rules?

In a sense an intelligent agent is similar to a stone rolling down a hill, both are moving towards a sort of equilibrium. The difference is that intelligence is following more complex trajectories as its ability to read and respond to environmental cues is vastly greater than that of a stone. Yet intelligent or not, the environment in which an agent is embedded plays a crucial role. There exist a fundamental dependency on unintelligent processes. Our environment is structured in such a way that we use information within it as an extension of our minds. The environment enables us to learn and improve our predictions by providing a testbed and a constant stream of data.

Necessary resources for an intelligence explosion

If artificial general intelligence is unable to seize the resources necessary to undergo explosive recursive self-improvement then the ability and cognitive flexibility of superhuman intelligence in and of itself, as characteristics alone, would have to be sufficient to self-modify its way up to massive superhuman intelligence within a very short time.

Without advanced real-world nanotechnology it will be considerable more difficult for an AGI to undergo quick self-improvement. It will have to make use of existing infrastructure, e.g. buy stocks of chip manufactures and get them to create more or better CPU’s. It will have to rely on puny humans for a lot of tasks. It won’t be able to create new computational substrate without the whole economy of the world supporting it. It won’t be able to create an army of robot drones overnight without it either.

Doing so it would have to make use of considerable amounts of social engineering without its creators noticing it. But, more importantly, it will have to make use of its existing intelligence to do all of that. The AGI would have to acquire new resources slowly, as it couldn’t just self-improve to come up with faster and more efficient solutions. In other words, self-improvement would demand resources. The AGI could not profit from its ability to self-improve regarding the necessary acquisition of resources to be able to self-improve in the first place.

Therefore the absence of advanced nanotechnology constitutes an immense blow to the possibility of explosive recursive self-improvement and risks from AI in general.

One might argue that an AGI will solve nanotechnology on its own and find some way to trick humans into manufacturing a molecular assembler and grant it access to it. But this might be very difficult.

There is a strong interdependence of resources and manufacturers. The AGI won’t be able to simply trick some humans to build a high-end factory to create computational substrate, let alone a molecular assembler. People will ask questions and shortly after get suspicious. Remember, it won’t be able to coordinate a world-conspiracy, it hasn’t been able to self-improve to that point yet because it is still trying to acquire enough resources, which it has to do the hard way without nanotech.

Anyhow, you’d probably need a brain the size of the moon to effectively run and coordinate a whole world of irrational humans by intercepting their communications and altering them on the fly without anyone freaking out.

People associated with the SIAI would at this point claim that if the AI can’t make use of nanotechnology it might make use of something we haven’t even thought about. But what, magic?

Artificial general intelligence, a single break-through?

Another point to consider when talking about risks from AI is how quickly the invention of artificial general intelligence will take place. What evidence do we have that there is some principle that, once discovered, allows us to grow superhuman intelligence overnight?

If the development of AGI takes place slowly, a gradual and controllable development, we might be able to learn from small-scale mistakes while having to face other risks in the meantime. This might for example be the case if intelligence can not be captured by a discrete algorithm, or is modular, and therefore never allow us to reach a point where we can suddenly build the smartest thing ever that does just extend itself indefinitely.

To me it doesn’t look like that we will come up with artificial general intelligence quickly, but rather that we will have to painstakingly optimize our expert systems step by step over long periods of times.

Paperclip maximizers

It is claimed that an artificial general intelligence might wipe us out inadvertently while undergoing explosive recursive self-improvement to more effectively pursue its terminal goals. I think that it is unlikely that most AI designs will not hold.

I agree with the argument that any AGI that isn’t made to care about humans won’t care about humans. But I also think that the same argument applies for spatio-temporal scope boundaries and resource limits. Even if the AGI is not told to hold, e.g. compute as many digits of Pi as possible, I consider it an far-fetched assumption that any AGI intrinsically cares to take over the universe as fast as possible to compute as many digits of Pi as possible. Sure, if all of that are presuppositions then it will happen, but I don’t see that most of all AGI designs are like that. Most that have the potential for superhuman intelligence, but who are given simple goals, will in my opinion just bob up and down as slowly as possible.

Complex goals need complex optimization parameters (the design specifications of the subject of the optimization process against which it will measure its success of self-improvement).

Even the creation of paperclips is a much more complex goal than telling an AI to compute as many digits of Pi as possible.

For an AGI, that was designed to design paperclips, to pose an existential risk, its creators would have to be capable enough to enable it to take over the universe on its own, yet forget, or fail to, define time, space and energy bounds as part of its optimization parameters. Therefore, given the large amount of restrictions that are inevitably part of any advanced general intelligence, the nonhazardous subset of all possible outcomes might be much larger than that where the AGI works perfectly yet fails to hold before it could wreak havoc.

Fermi paradox

The Fermi paradox does allow for and provide the only conclusions and data we can analyze that amount to empirical criticism of concepts like that of a Paperclip maximizer and general risks from superhuman AI’s with non-human values without working directly on AGI to test those hypothesis ourselves.

If you accept the premise that life is not unique and special then one other technological civilisation in the observable universe should be sufficient to leave potentially observable traces of technological tinkering.

Due to the absence of any signs of intelligence out there, especially paper-clippers burning the cosmic commons, we might conclude that unfriendly AI could not be the most dangerous existential risk that we should worry about.

Summary

In principle we could build antimatter weapons capable of destroying worlds, but in practise it is much harder to accomplish.

There are many question marks when it comes to the possibility of superhuman intelligence, and many more about the possibility of recursive self-improvement. Most of the arguments in favor of those possibilities solely derive their appeal from being vague.

Further reading

• Intelligence Explosion - A Disjunctive or Conjunctive Event?
• The Hanson-Yudkowsky AI-Foom Debate
• The Betterness Explosion
• Is The City-ularity Near?
• How far can AI jump?
• Why I’m Not Afraid of the Singularity
• What’s the Likelihood of the Singularity? Part One: Artificial Intelligence
• When Exactly Will Computers Go Ape-Shi* and Take Over?
• The slowdown hypothesis (extended abstract)
• The singularity as faith (extended abstract)

(The following is a summary of some of my previous submissions that I originally created for my personal blog.)

...an intelligence explosion may have fair probability, not because it occurs in one particular detailed scenario, but because, like the evolution of eyes or the emergence of markets, it can come about through many different paths and can gather momentum once it gets started. Humans tend to underestimate the likelihood of such “disjunctive” events, because they can result from many different paths (Tversky and Kahneman 1974). We suspect the considerations in this paper may convince you, as they did us, that this particular disjunctive event (intelligence explosion) is worthy of consideration.

— lukeprog, Intelligence Explosion analysis draft: introduction

It seems to me that all the ways in which we disagree have more to do with philosophy (how to quantify uncertainty; how to deal with conjunctions; how to act in consideration of low probabilities) [...] we are not dealing with well-defined or -quantified probabilities. Any prediction can be rephrased so that it sounds like the product of indefinitely many conjunctions. It seems that I see the “SIAI’s work is useful scenario” as requiring the conjunction of a large number of questionable things [...]

— Holden Karnofsky, 6/24/11 (GiveWell interview with major SIAI donor Jaan Tallinn, PDF)

Disjunctive arguments

People associated with the Singularity Institute for Artificial Intelligence (SIAI) like to claim that the case for risks from AI is supported by years worth of disjunctive lines of reasoning. This basically means that there are many reasons to believe that humanity is likely to be wiped out as a result of artificial general intelligence. More precisely it means that not all of the arguments supporting that possibility need to be true, even if all but one are false risks from AI are to be taken seriously.

The idea of disjunctive arguments is formalized by what is called a logical disjunction. Consider two declarative sentences, A and B. The truth of the conclusion (or output) that follows from the sentences A and B does depend on the truth of A and B. In the case of a logical disjunction the conclusion of A and B is only false if both A and B are false, otherwise it is true. Truth values are usually denoted by 0 for false and 1 for true. A disjunction of declarative sentences is denoted by OR or ∨ as an infix operator. For example, (A(0)∨B(1))(1), or in other words, if statement A is false and B is true then what follows is still true because statement B is sufficient to preserve the truth of the overall conclusion.

Generally there is no problem with disjunctive lines of reasoning as long as the conclusion itself is sound and therefore in principle possible, yet in demand of at least one of several causative factors to become actual. I don’t perceive this to be the case for risks from AI. I agree that there are many ways in which artificial general intelligence (AGI) could be dangerous, but only if I accept several presuppositions regarding AGI that I actually dispute.

By presuppositions I mean requirements that need to be true simultaneously (in conjunction). A logical conjunction is only true if all of its operands are true. In other words, the a conclusion might require all of the arguments leading up to it to be true, otherwise it is false. A conjunction is denoted by AND or ∧.

Now consider the following prediction: <Mary is going to buy one of thousands of products in the supermarket.>

The above prediction can be framed as a disjunction: Mary is going to buy one of thousands of products in the supermarket, 1.) if she is hungry 2.) if she is thirsty 3.) if she needs a new coffee machine. Only one of the 3 given possible arguments need to be true in order to leave the overall conclusion to be true, that Mary is going shopping. Or so it seems.

The same prediction can be framed as a conjunction: Mary is going to buy one of thousands of products in the supermarket 1.) if she has money 2.) if she has some needs 3.) if the supermarket is open. All of the 3 given factors need to be true in order to render the overall conclusion to be true.

That a prediction is framed to be disjunctive does not speak in favor of the possibility in and of itself. I agree that it is likely that Mary is going to visit the supermarket if I accept the hidden presuppositions. But a prediction is only at most as probable as its basic requirements. In this particular case I don’t even know if Mary is a human or a dog, a factor that can influence the probability of the prediction dramatically.

The same is true for risks from AI. The basic argument in favor of risks from AI is that of an intelligence explosion, that intelligence can be applied to itself in an iterative process leading to ever greater levels of intelligence. In short, artificial general intelligence will undergo explosive recursive self-improvement.

Hidden complexity

Explosive recursive self-improvement is one of the presuppositions for the possibility of risks from AI. The problem is that this and other presuppositions are largely ignored and left undefined. All of the disjunctive arguments put forth by the SIAI are trying to show that there are many causative factors that will result in the development of unfriendly artificial general intelligence. Only one of those factors needs to be true for us to be wiped out by AGI. But the whole scenario is at most as probable as the assumption hidden in the words <artificial general intelligence> and <explosive recursive self-improvement>.

<Artificial General Intelligence> and <Explosive Recursive Self-improvement> might appear to be relatively simple and appealing concepts. But most of this superficial simplicity is a result of the vagueness of natural language descriptions. Reducing the vagueness of those concepts by being more specific, or by coming up with technical definitions of each of the words they are made up of, reveals the hidden complexity that is comprised in the vagueness of the terms.

If we were going to define those concepts and each of its terms we would end up with a lot of additional concepts made up of other words or terms. Most of those additional concepts will demand explanations of their own made up of further speculations. If we are precise then any declarative sentence (P#) (all of the terms) used in the final description will have to be true simultaneously (P#∧P#). And this does reveal the true complexity of all hidden presuppositions and thereby influence the overall probability, P(risks from AI) = P(P1∧P2∧P3∧P4∧P5∧P6∧…). That is because the conclusion of an argument that is made up of a lot of statements (terms) that can be false is more unlikely to be true since complex arguments can fail in a lot of different ways. You need to support each part of the argument that can be true or false and you can therefore fail to support one or more of its parts, which in turn will render the overall conclusion false.

To summarize: If we tried to pin down a concept like <Explosive Recursive Self-Improvement> we would end up with requirements that are strongly conjunctive.

Making numerical probability estimates

But even if the SIAI was going to thoroughly define those concepts, there is still more to the probability of risks from AI than the underlying presuppositions and causative factors. We also have to integrate our uncertainty about the very methods we used to come up with those concepts, definitions and our ability to make correct predictions about the future and integrate all of it into our overall probability estimates.

Take for example the following contrived quote:

We have to take over the universe to save it by making the seed of an artificial general intelligence, that is undergoing explosive recursive self-improvement, extrapolate the coherent volition of humanity, while acausally trading with other superhuman intelligences across the multiverse.

Although contrived, the above quote does only comprise actual beliefs hold by people associated with the SIAI. All of those beliefs might seem somewhat plausible inferences and logical implications of speculations and state of the art or bleeding edge knowledge of various fields. But should we base real-life decisions on those ideas, should we take those ideas seriously? Should we take into account conclusions whose truth value does depend on the conjunction of those ideas? And is it wise to make further inferences on those speculations?

Let’s take a closer look at the necessary top-level presuppositions to take the above quote seriously:

1. The many-worlds interpretation
2. Belief in the Implied Invisible
3. Timeless Decision theory
4. Intelligence explosion

1: Within the lesswrong/SIAI community the many-worlds interpretation of quantum mechanics is proclaimed to be the rational choice of all available interpretations. How to arrive at this conclusion is supposedly also a good exercise in refining the art of rationality.

2: P(Y|X) ≈ 1, then P(X∧Y) ≈ P(X)

In other words, logical implications do not have to pay rent in future anticipations.

3: “Decision theory is the study of principles and algorithms for making correct decisions—that is, decisions that allow an agent to achieve better outcomes with respect to its goals.”

4: “Intelligence explosion is the idea of a positive feedback loop in which an intelligence is making itself smarter, thus getting better at making itself even smarter. A strong version of this idea suggests that once the positive feedback starts to play a role, it will lead to a dramatic leap in capability very quickly.”

To be able to take the above quote seriously you have to assign a non-negligible probability to the truth of the conjunction of #1,2,3,4, 1∧2∧3∧4. Here the question is not not only if our results are sound but if the very methods we used to come up with those results are sufficiently trustworthy. Because any extraordinary conclusions that are implied by the conjunction of various beliefs might outweigh the benefit of each belief if the overall conclusion is just slightly wrong.

Not enough empirical evidence

Don’t get me wrong, I think that there sure are convincing arguments in favor of risks from AI. But do arguments suffice? Nobody is an expert when it comes to intelligence. My problem is that I fear that some convincing blog posts written in natural language are simply not enough.

Just imagine that all there was to climate change was someone who never studied the climate but instead wrote some essays about how it might be physical possible for humans to cause a global warming. If the same person then goes on to make further inferences based on the implications of those speculations, am I going to tell everyone to stop emitting CO2 because of that? Hardly!

Or imagine that all there was to the possibility of asteroid strikes was someone who argued that there might be big chunks of rocks out there which might fall down on our heads and kill us all, inductively based on the fact that the Earth and the moon are also a big rocks. Would I be willing to launch a billion dollar asteroid deflection program solely based on such speculations? I don’t think so.

Luckily, in both cases, we got a lot more than some convincing arguments in support of those risks.

Another example: If there were no studies about the safety of high energy physics experiments then I might assign a 20% chance of a powerful particle accelerator destroying the universe based on some convincing arguments put forth on a blog by someone who never studied high energy physics. We know that such an estimate would be wrong by many orders of magnitude. Yet the reason for being wrong would largely be a result of my inability to make correct probability estimates, the result of vagueness or a failure of the methods I employed to come up with those estimates. The reason for being wrong by many orders of magnitude would have nothing to do with the arguments in favor of the risks, as they might very well be sound given my epistemic state and the prevalent uncertainty.

I believe that mere arguments in favor of one risk do not suffice to neglect other risks that are supported by other kinds of evidence. I believe that logical implications of sound arguments should not reach out indefinitely and thereby outweigh other risks whose implications are fortified by empirical evidence. Sound arguments, predictions, speculations and their logical implications are enough to demand further attention and research, but not much more.

Logical implications

Artificial general intelligence is already an inference made from what we currently believe to be true, going a step further and drawing further inferences from previous speculations, e.g. explosive recursive self-improvement, is in my opinion a very shaky business.

What would happen if we were going to let logical implications of vast utilities outweigh other concrete near-term problems that are based on empirical evidence? Insignificant inferences might exhibit hyperbolic growth in utility: 1.) There is no minimum amount of empirical evidence necessary to extrapolate the expected utility of an outcome. 2.) The extrapolation of counterfactual alternatives is unbounded, logical implications can reach out indefinitely without ever requiring new empirical evidence.

Hidden disagreement

All of the above hints at a general problem that is the reason for why I think that discussions between people associated with the SIAI, its critics and those who try to evaluate the SIAI, won’t lead anywhere. Those discussions miss the underlying reason for most of the superficial disagreement about risks from AI, namely that there is no disagreement about risks from AI in and of itself.

There are a few people who disagree about the possibility of AGI in general, but I don’t want to touch on that subject in this post. I am trying to highlight the disagreement between the SIAI and people who accept the notion of artificial general intelligence. With regard to those who are not skeptical of AGI the problem becomes more obvious when you turn your attention to people like John Baez organisations like GiveWell. Most people would sooner question their grasp of “rationality” than give five dollars to a charity that tries to mitigate risks from AI because their calculations claim it was “rational” (those who have read the article by Eliezer Yudkowsky on ‘Pascal’s Mugging‘ know that I used a statement from that post and slightly rephrased it). The disagreement all comes down to a general averseness to options that have a low probability of being factual, even given that the stakes are high.

Nobody is so far able to beat arguments that bear resemblance to Pascal’s Mugging. At least not by showing that it is irrational to give in from the perspective of a utility maximizer. One can only reject it based on a strong gut feeling that something is wrong. And I think that is what many people are unknowingly doing when they argue against the SIAI or risks from AI. They are signaling that they are unable to take such risks into account. What most people mean when they doubt the reputation of people who claim that risks from AI need to be taken seriously, or who say that AGI might be far off, what those people mean is that risks from AI are too vague to be taken into account at this point, that nobody knows enough to make predictions about the topic right now.

When GiveWell, a charity evaluation service, interviewed the SIAI (PDF), they hinted at the possibility that one could consider the SIAI to be a sort of Pascal’s Mugging:

GiveWell: OK. Well that’s where I stand – I accept a lot of the controversial premises of your mission, but I’m a pretty long way from sold that you have the right team or the right approach. Now some have argued to me that I don’t need to be sold – that even at an infinitesimal probability of success, your project is worthwhile. I see that as a Pascal’s Mugging and don’t accept it; I wouldn’t endorse your project unless it passed the basic hurdles of credibility and workable approach as well as potentially astronomically beneficial goal.

This shows that lot of people do not doubt the possibility of risks from AI but are simply not sure if they should really concentrate their efforts on such vague possibilities.

Technically, from the standpoint of maximizing expected utility, given the absence of other existential risks, the answer might very well be yes. But even though we believe to understand this technical viewpoint of rationality very well in principle, it does also lead to problems such as Pascal’s Mugging. But it doesn’t take a true Pascal’s Mugging scenario to make people feel deeply uncomfortable with what Bayes’ Theorem, the expected utility formula, and Solomonoff induction seem to suggest one should do.

Again, we currently have no rational way to reject arguments that are framed as predictions of worst case scenarios that need to be taken seriously even given a low probability of their occurrence due to the scale of negative consequences associated with them. Many people are nonetheless reluctant to accept this line of reasoning without further evidence supporting the strong claims and request for money made by organisations such as the SIAI.

Here is what mathematician and climate activist John Baez has to say:

Of course, anyone associated with Less Wrong would ask if I’m really maximizing expected utility. Couldn’t a contribution to some place like the Singularity Institute of Artificial Intelligence, despite a lower chance of doing good, actually have a chance to do so much more good that it’d pay to send the cash there instead?

And I’d have to say:

1) Yes, there probably are such places, but it would take me a while to find the one that I trusted, and I haven’t put in the work. When you’re risk-averse and limited in the time you have to make decisions, you tend to put off weighing options that have a very low chance of success but a very high return if they succeed. This is sensible so I don’t feel bad about it.

2) Just to amplify point 1) a bit: you shouldn’t always maximize expected utility if you only live once. Expected values — in other words, averages — are very important when you make the same small bet over and over again. When the stakes get higher and you aren’t in a position to repeat the bet over and over, it may be wise to be risk averse.

3) If you let me put the $100,000 into my retirement account instead of a charity, that’s what I’d do, and I wouldn’t even feel guilty about it. I actually think that the increased security would free me up to do more risky but potentially very good things!

All this shows that there seems to be a fundamental problem with the formalized version of rationality. The problem might be human nature itself, that some people are unable to accept what they should do if they want to maximize their expected utility. Or we are missing something else and our theories are flawed. Either way, to solve this problem we need to research those issues and thereby increase the confidence in the very methods used to decide what to do about risks from AI, or to increase the confidence in risks from AI directly, enough to make it look like a sensible option, a concrete and discernable problem that needs to be solved.

Many people perceive the whole world to be at stake, either due to climate change, war or engineered pathogens. Telling them about something like risks from AI, even though nobody seems to have any idea about the nature of intelligence, let alone general intelligence or the possibility of recursive self-improvement, seems like just another problem, one that is too vague to outweigh all the other risks. Most people feel like having a gun pointed to their heads, telling them about superhuman monsters that might turn them into paperclips then needs some really good arguments to outweigh the combined risk of all other problems.

But there are many other problems with risks from AI. To give a hint at just one example: if there was a risk that might kill us with a probability of .7 and another risk with .1 while our chance to solve the first one was .0001 and the second one .1, which one should we focus on? In other words, our decision to mitigate a certain risk should not only be focused on the probability of its occurence but also on the probability of success in solving it. But as I have written above I believe that the most pressing issue is to increase the confidence into making decisions under extreme uncertainty or to reduce the uncerainty itself.

The following is a clipping of a documentary about transhumanism that I recorded when it aired on Arte, September 22 2011.

At the beginning and end of the video Luke Muehlhauser and Michael Anissimov give a short commentary.

Download here: German, French (ask for HD download link). Should play with VLC player.

Sadly, the people who produced the show seemed to be somewhat confused about the agenda of the Singularity Institute. At one point they seem to be saying that the SIAI believes into "the good in the machines", adding "how naive!", while the next sentence talks about how the SIAI tries to figure out how to make machines respect humans.

Here is the original part of the clip that I am talking about:

In San Francisco glaubt eine Vereinigung ehrenamtlicher junger Wissenschaftler dennoch an das Gute im Roboter. Wie naiv! Hier im Singularity Institute, dass Kontakte zu den großen Unis wie Oxford hat, zerbricht man sich den Kopf darüber, wie man zukünftigen Formen künstlicher Intelligenz beibringt, den Menschen zu respektieren.

Die Forscher kombinieren Daten aus Informatik und psychologischen Studien. Ihr Ziel: Eine Not-to-do-Liste, die jedes Unternehmen bekommt, das an künstlicher Intelligenz arbeitet.

My translation:

In San Francisco however, a society of young voluntary scientists believes in the good in robots. How naive! Here at the Singularity Institute, which has a connection to big universities like Oxford, they think about how to teach future artificial intelligences to respect humans.

I am a native German speaker by the way, maybe someone else who speaks German can make more sense of it (and is willing to translate the whole clip).

After reading the FHI achievement report for 2008-2010, I thought it might be useful to compare their achievements to those of SIAI during the same time period. Since SIAI does not have an equivalent report, I've mostly pulled the data of their achievements from the SIAI blog.

My intention here is to help figure out which organization makes better use of my donations. For that purpose, I'm only looking at actual concrete outputs, and ignoring achievements such as successful fundraising drives or the hiring of extra staff.

For citation counts, I'm using Google Scholar data as-is. Note that this will include both self-cites and some cites from pages that really shouldn't be counted, since Google Scholar seems to be a bit liberal about what it includes in its database. I'm unsure as to whether or not the citation counts are very meaningful, since there hasn't been much time for anyone to cite papers published in 2010, say. But I'm including them anyway.

Future of Humanity Institute

Publications. The Achievement Report highlights three books and 22 journal articles. In addition, FHI staff has written 34 book chapters for academic volumes, including Companion to Philosophy of Technology; New Waves in Philosophy of Technology; Philosophy: Theoretical and Empirical Explorations; and Oxford Handbook of Neuroethics.

The three are the hardcover and paperback editions of Human Enhancement, as well as a paperback edition of Anthropic Bias: Observation Selection Effects in Science and Philosophy. Human Enhancement has been cited 22 times. Anthropic Bias was originally published in 2002, so I'm not including its citation count.

The highlighted 22 journal articles had been cited 59 times in total. The overwhelmingly most cited article was Cognitive Enhancement: Methods, Ethics, Regulatory Challenges in Science and Engineering Ethics, with 39 cites. The runner-up was Probing the Improbable: Methodological. Challenges for Risks with Low Probabilities and High Stake, with 5 cites. The remaining articles had 0-3 cites. But while Cognitive Enhancement is listed as a 2009 paper, it's worth noting that the first draft version of it was posted on Nick Bostrom's website back in 2006, and it has had time to accumulate cites since then. If we exclude it, FHI's 2008-2010 papers have been cited 20 times.

It's not listed in the Achievement Report, but I also want to include the 2008 Whole Brain Emulation Roadmap, which has been cited 15 times, bringing the total count (excluding Cognitive Enhancement) to 35.

Presentations. FHI members have given a total of 95 invited lectures and conference presentations.

Media appearances. Some 100 media appearances, including print, radio, and television appearances, since January 2009. These include BBC television, New Scientist, National Geographic, The Guardian, ITV, Bloomberg News, Discovery Channel, ABC, Radio Slovenia, Wired Magazine, BBC world service, Volkskrant (German newspaper), Utbildningsradion (Swedish national radio), Mehr News Agency (Iranian), Mladina Weekly (Slovenian magazine), Jyllands-Posten and Weekenavisen (Danish newspapers), Bayerisher Rundfunk (German radio), The History Channel, O Estado de São Paulo (Brazillian newspaper), Euronews, Kvallsposten (Swedish newspaper), City Helsinki (Finnish radio), Focus, Dutch Film and Television Academy, The Smart Manager (Indian magazine), Il Sole 24 Ore (Italian monthly), The Bulletin of the Atomic Sciences, Time Magazine, Astronomy Now, and Radio Bar-Kulan (Kenya).

Visitors. "The Institute receives many requests from students and scholars who wish to visit the Institute, only a few of which are accepted because of capacity limitations. The FHI has hosted a number of distinguished academic visitors over the past two years within its various areas of activity, such as Profs. David Chalmers, Michael Oppenheimer, and Thomas Homer-Dixon."

Policy advice. The Achievement Report highlights 23 groups or events which have received policy advice from either Nick Bostrom or Anders Sandberg. These include the World Economic Forum, the Public Services Offices of the Prime Minister's Office of Singapore, the UK Home Office, If (Stockholm insurance company), Jane Street Capital, IARPA (Intelligence Advanced Research Projects Activity) for US Government, The Swedish Institute for Infectious Disease Control, and setting up a research network, "A differential view of enhancement", within the Volkswagen Foundation.

Organized events. Three organized events. 1: Cognitive Enhancement Workshop. 2: Symposium on cognitive enhancement and related ethical and policy issues. 3: Uncertainty, Lags and Nonlinearity: Challenges to governance in a turbulent world.

Singularity Institute

Publications. The SIAI publications page has 15 papers from the 2008-2010 period, of which 11 are listed under "recent publications", 1 under "software", and 3 under "talks and working papers". Of these, Superintelligence does not imply benevolence has been cited once. The rest all have no citations.

The Sequences were written during this time period. They consist of about a million words, and might very well have a bigger impact than all the other FHI and SIAI articles together - though that's very hard to quantify.

Presentations and Media Appearances. The SIAI blog mentions a number of media appearances and presentations at various venues, but I don't have the energy to go through them all and count. From a quick eyeballing of the blog, though, SIAI has nowhere near as many presentations and media appearances as FHI.

Visitors. The Visiting Fellows page has a list of 27 Visiting Fellows from around the world, who attend or hold degrees from universities including Harvard, Stanford, Yale, Cambridge, Carnegie Mellon, Auckland University, Moscow Institute of Physics and Technology, and the University of California-Santa Barbara

Online communities and tools. Less Wrong was founded in 2009, and Google Analytics says that by the end of 2010, it had had over a million unique visitors.

Note that LW is an interesting case: as an FHI/SIAI collaboration, both organizations claim credit for it. However, since LW is to such a huge extent Eliezer's creation, and I'm not sure of what exactly the FHI contribution to LW is, I'm counting it as an SIAI and not a joint achievement.

SIAI also created the Uncertain Future, a tool for estimating the probability of AI.

Organized events. SIAI held Singularity Summits on all three years. The first Singularity Summit Australia was held in 2010. In 2008, SIAI co-sponsored the Converge unconference.

Ben Goertzel, acting as the SIAI Director of Research at the time, organized the 2008 and 2009 conferences on Artificial General Intelligence. He also co-organized a 2009 workshop on machine consciousness,

Artificial Intelligence projects. SIAI provided initial funding for the OpenCog project, as well as sponsoring Google Summer of Code events relating to the project in 2008 and 2009.

Overall

Based on this data, which organization is more deserving of my money? Hard to say, especially since SIAI has been changing a lot. The general AGI research, for instance, isn't really something that's being pursued anymore, and Ben Goertzel is no longer with the organization. Eliezer is no longer writing the sequences, which were possibly the biggest SIAI achievement of the whole 2008-2010 period.

Still, FHI's accomplishments seem a lot more impressive overall, suggesting that they might be a better target for the money. On the other hand, they are not as tightly focused on AI as SIAI is.

One imporant question is also the amount of funding the two organizations have had: accomplishing a lot is easier if you have more money. If an organization has thrice as much money, they should be expected to achieve thrice as much. SIAI's revenue was $426,000 in 2008 and $628,000 in 2009. FHI's funding was around $711,000 for 10/2008 - 10/2009. I don't know the 2010 figure for either organization. The FHI report also says the following:

To appreciate the significance of what has been accomplished, it should be kept in mind that the FHI has been understaffed for much of this period. One of our James Martin Research Fellows, Dr Rebecca Roache, has been on maternity leave for the past year. Our newest James Martin Research Fellow, Dr Eric Mandelbaum, who was recruited from an extremely strong field of over 170 applicants, has been in post for only two months. Thus, for half of the two-year period, FHI’s research staff has consisted of two persons, Professor Nick Bostrom and Dr Anders Sandberg.

SIAI - An Examination notes that in both 2008 and 2009, SIAI paid salaries to three people, so for a while at least, the amount of full-time staff in the two organizations was roughly comparable.

After just spending some time browsing free nonficton kindle ebooks on Amazon, it occurred to me that it might be a good idea for SIAI/LW to publish for free download through Amazon some introductory LW essays and other useful introductory works like Twelve Virtues of Rationality and The Simple Truth.

People who search for 'rationality' on Google will see Eliezer's Twelve Virtues of Rationality and LW. It would nice if searching for rationality on Amazon also led people to similar resources that could be read on the Kindle with just one click. It would considerably expand the audience of potential readers (and LW contributors and SIAI donors).

The $125,000, 2011 Summer Matching Challenge was a success! We met our goal 2 days early, raising $250,000 total for Singularity Institute operations. Here is the blog post announcement.

I have seen too many discussions of Friendly AI, here and elsewhere (e.g. in comments at Michael Anissimov's blog), detached from any concrete idea of how to do it. Sometimes the issue is the lack of code, demos, or a practical plan from SIAI. SIAI is seen as a source of wishful thinking about magic machines that will solve all our problems for us, or as a place engaged in a forever quest for nebulous mathematical vaporware such as "reflective decision theory". You will get singularity enthusiasts who say, it's great that SIAI has given the concept of FAI visibility, but enough with the philosophy, let's get coding! ... does anyone know where to start? And you will get singularity skeptics who say, unfriendly AI is a bedtime ghost story for credulous SF fans, wake me up when SIAI actually ships a product. Or, within this subculture of rationalist altruists who want to do the optimal thing, you'll get people saying, I don't know if I should donate, because I don't see how any of this is supposed to happen.

So in this post I want to sketch what a "practical" plan for Friendly AI looks like. I'm not here to advocate this plan - I'm not saying this is the right way to do it. I'm just providing an example of a plan that could be pursued in the real world. Perhaps it will also allow people to better understand SIAI's indirect approach.

I won't go into the details of financial or technical logistics. If we were talking about how to get to the moon from Earth, then the following plan is along the lines of "Make a chemical-powered rocket big enough to get you there." Once you have that concept, you still have a lot of work to do, but you are at least on the right track - compared to people who want to make a teleportation device or a balloon that goes really high. But I will make one remark about how the idea of Friendly AI is framed. At present, it is discussed in conjunction with a whole cornucopia of science fiction notions such as: immortality, conquering the galaxy, omnipresent wish-fulfilling super-AIs, good and bad Jupiter-brains, mind uploads in heaven and hell, and so on. Similarly, we have all these thought-experiments: guessing games with omniscient aliens, decision problems in a branching multiverse, "torture versus dust specks". Whatever the ultimate relevance of such ideas, it is clearly possible to divorce the notion of Friendly AI from all of them. If a FAI project was trying to garner mass support, it first needs to be comprehensible, and the simple approach would be to say it is simply an exercise in creating artificial intelligence that does the right thing. Nothing about utopia; nothing about dystopia caused by unfriendly AI; nothing about godlike superintelligence; just the scenario, already familiar in popular culture, of robots, androids, computers you can talk with. All that is coming, says the practical FAI project, and we are here to design these new beings so they will be good citizens, a positive rather than a negative addition to the world.

So much for how the project describes itself to the world at large. What are its guiding technical conceptions? What's the specific proposal which will allow educated skeptics to conclude that this might get off the ground? Remember that there are two essential challenges to overcome: the project has to create intelligence, and it has to create ethical intelligence; what we call, in our existing discussions, "AGI" - artificial general intelligence - and "FAI" - friendly artificial intelligence.

There is a very simple approach which - like the idea of a chemical-powered rocket which gets you to the moon - should be sufficient to get you to FAI, when sufficiently elaborated. It can be seen by stripping away some of the complexities peculiar to SIAI's strategy, complexities which tend to dominate the discussion. The basic idea should also be thoroughly familiar. We are to conceive of the AI as having two parts, a goal system and a problem-solving system. AGI is achieved by creating a problem-solving system of sufficient power and universality; FAI is achieved by specifying the right goal system.

SIAI, in discussing the quest for the right goal system, emphasizes the difficulties of this process and the unreliability of human judgment. Their idea of a solution is to use artificial intelligence to neuroscientifically deduce the actual algorithmic structure of human decision-making, and to then employ a presently nonexistent branch of decision theory to construct a goal system embodying ideals implicit in the unknown human cognitive algorithms.

The practical approach would not bother with this attempt to outsource the task of designing the AI's morality, to a presently nonexistent neuromathematical cognitive bootstrap process. While fully cognizant of the fact that value is complex, as eloquently attested by Eliezer in many speeches, the practical FAI project would nonetheless choose the AI's goal system in the old-fashioned way, by human deliberation and consensus. You would get a team of professional ethicists, some worldly people like managers, some legal experts in the formulation of contracts, and together you would hammer out a mission statement for the AI. Then you would get your programmers and your cognitive scientists to implement that goal condition in a way such that the symbols have the meanings that they are supposed to have. End of story.

So far, all we've done is to make a wish. We've decided, after appropriate deliberation, what to wish for, and we have found a way to represent it in symbols. All that means nothing if we can't create AGI, the problem solver with at least a human level of intelligence. Here again, SIAI comes in for a lot of criticism, from two angles: it's said to have no ideas about how to create AGI, and it's said to actively discourage work on AGI, on the grounds that we need to solve the FAI problem first. Instead, it only discusses hopelessly impractical models of cognition like AIXI and exact Bayesian inference, that are mostly of theoretical interest.

Our practical FAI project has "solved" FAI by simply coming to an agreement on what to wish for, and by studying with legalistic care how to avoid pitfalls and loopholes in the finer details of the wish; but what is its approach to the hard technical problem of AGI? The answer is, first of all, heuristics and incremental improvement. Projects like Lenat's Cyc are on the right track. A newborn AI has to be seeded with useful knowledge, including useful knowledge of problem-solving methods. It doesn't have time to discover such things entirely unaided. We should not imagine AGI developing just from a simple architecture, like Schmidhuber's Gödel machine, but from a basic architecture plus a large helping of facts and heuristics which are meant to give it a head start.

So fine, the practical approach to AGI isn't a search for a single killer concept, it's a matter of incrementally increasing the power of a general-purpose problem solver with many diverse ingredients in its design, so that it becomes more and more capable and independent. Ben Goertzel's approach to AGI exhibits the sort of eclectic pluralism that I have in mind. Still, we do need a selling point, something which shows that we're different, that we're aiming for the stars and we have a plan to get there.

Here, I want to use Steve Omohundro's paper "The Basic AI Drives" in a slightly unusual way. The paper lists a number of behaviors that should be exhibited by a sufficiently sophisticated AI: it will try to model its own operation, clarify its goals, protect them from modification, protect itself from destruction, acquire resources and use them efficiently... The twist I propose is that Omohundro's list of drives should be used as a design specification. If your goal is AGI, then you want a cognitive architecture that will exhibit these emergent behaviors. They offer a series of milestones for your theorists and developers: a criterion of progress, and a set of intermediate goals sufficient to bridge the gap between a blank-slate beginning and an open-ended problem solver.

That's the whole plan. It's an anticlimax, I know, for anyone who might have imagined that there was a magic formula for superintelligence coming at the end of this post. But I do claim that what I have described is the skeleton of a plan which can be fleshed out, and which, if it was fleshed out and pursued, would produce goal-directed AGI. Whether the project as I have described it would really produce "friendly" AI is another matter. Anyone versed in the folk wisdom about FAI should be able to point out multiple points of potential failure. But I hope this makes it a little clearer, to people who just don't see how FAI is supposed to happen at all, how it might be pursued in the real world.

I've been having a bit of a back-and-forth with a friend about what appears to be a charisma problem with the SIAI, and was hoping you lovely folks had thoughts on the matter. My friend was going through the Eliezer Q&A videos, specifically Question #7, "What's your advice for Less Wrong readers who want to help save the human race?" He typed up a transcript for Eliezer's answer, and went on to say:

Now, I freely admit that he is talking extemporaneously.  That he
maybe is giving point-by-point details, interpreting the question as a
laundry-list request of job opportunities, and that criticizing with a
call for brevity is easy as a response rather than a first try, but
HOLY SHIT, COME ON!  REALLY!? [...]

All that aside, here's how a, ahem, human, might respond quickly and
clearly to the question:

Doing what you like and are most efficient at (for money) is the best
way to get resources to us if you support our cause.  Make money at
those things and send it to us if think we're worth it.

Done.

He went on to mention that he really likes Eliezer's writings, and that his issue rests with the verbal skills of SIAI's leadership, not with the quality of their works.

I replied:

On the one hand, it would be extremely beneficial for them to get some kind of propaganda minister. On the other, I think that would signal to nerds like us that they are corrupt money-whores. If that is the case, they are stuck being stilted nerds if they want to attract brains, and if they want money, they're stuck watering down their fan base with Dan Brown readers or something.

I also suggested a couple possible (though rather outlandish) ways to make an organization wildly popular. Specifically, to hire a marketing researcher like Frank Luntz to figure out what talking points would win the hearts and minds of the greatest number of people, or alternately to get major brand loyalties by having a cult figure like Steve Jobs representing the SIAI. Of course, I am stating this much more eloquently than I did in the email.

His reply deserves full posting here (with his permission, of course):

I disagree with your proposition that getting a competent marketing
firm involved would suddenly create a contradistinction with the
organization proper.

From everything I've seen/read, these people are nothing if not fully
aware of the compartmentalized world we live in.  That this enterprise
requires a particular something upon another something with these
other bits running in the background.  Hell, in a grossly simplistic
interpretation, accomplishing this nested complexity is the whole of
their aim.

What I would say, however, is that the idea that these people are even
aware of any sort of nerd brand loyalty is entirely off-base.  I don't
think these folks operate with that realization in mind.  I think if
you even brought up the concept, they'd look at you askew in the same
way as telling [mutual acquaintance] he was a geek.  "But...I'm...what?  I'm
doing...cool things."

No, I think the fact that they haven't invested in marketing may be
mainly due to money woes, but more likely revealing a fatal flaw in
their infrastructure, in that their intricate understanding of what
they need to do ultimately fails to absorb themselves in the mix.
Failing to see their own operation as needing the societal locomotive
powers to get the final job done.

If that fear is true, we're in an awful spot indeed.  Needing to be
rescued by people ignorant of how to rescue themselves.

Let's hope it's the money woes, then.  Or...hmm...maybe a vacuum to be
met by someone who believes in the cause and also possesses mild
wordcraft?  What fancy!

The question is now open. Does SIAI have a PR problem? If so, is it due to finances, lack of talent, or something else? Is there an Eternal September issue with watering down the brand (would you support the SIAI if they started investing heavily in advertising campaigns, or would you get a bit suspicious?)? Should they pay Frank Luntz to figure out what transhumanism terms work best with your average family? My friend and I are dying to know.

(Motivated by this comment.)

The Singularity Institute's new logo, chosen from a competition, doesn't seem to be publicly available in SVG format.  But presumably, the winner delivered an SVG (or another vector format) instead of a JPEG.  The vector form should be made publicly available, like how Wikipedia's logo is available as an SVG here.

To start with, this could be used to improve the Singularity Institute page on Wikipedia, which is in dire need of a logo.

(I actually have no personal use for this, I'm simply a connoisseur of high-quality logos.)

Luke posted about this on SIAI blog. Essentially, you can get a credit card with cash-back rewards program that automatically donates money to Singularity Institute.

This seems to me like a dream come true. Am I missing something? Are there any catches? Is there a better rewards program, which I can use to save more money, so I can donate more money to SIAI?

GiveWell recently release notes from their interview with Jaan Tallinn, Skype co-founder and a major SIAI donor, about SIAI (link). Holden Karnofsky says 

[M]y key high-level takeaways are that

1. I appreciated Jaan's thoughtfulness and willingness to engage in depth. It was an interesting exchange.
2. I continue to disagree with the way that SIAI is thinking about the "Friendliness" problem. 
3. It seems to me that all the ways in which Jaan and I disagree on this topic have more to do with philosophy (how to quantify uncertainty; how to deal with conjunctions; how to act in consideration of low probabilities) and with social science-type intuitions (how would people likely use a particular sort of AI) than with computer science or programming (what properties has software usually had historically; which of these properties become incoherent/hard to imagine when applied to AGI)

My purpose in writing this is twofold.

First (chronologically: I thought of this earlier than the other), I want to discuss some of the pragmatic points in how I got myself to do it.

The most important thing is that I didn't try to force the decision through with willpower. Instead, I slipped it through with doublethink. I knew perfectly well - and have known for months - that giving money to SIAI was the right thing to do. But I didn't do it. I spent money on things like Minecraft instead.

But somehow I found myself at the donation page, and I didn't think about it. Or, rather, I didn't let myself think about the fact that I was thinking about it. I made a series of expected-value guesstimates aimed at working around my own cognitive limitations.

I chose monthly donation over one-time because $20 monthly sounds like about the same amount of money as $20; past experience with recurring donations suggests that I tend to leave automatic recurring donations in place for about a year or two, so that probably gained me about a factor of 20. Similarly, I chose $20 as the largest amount that wouldn't put me in serious risk of chickening out and not donating anything.

In order to pull this off, I had to avoid thinking certain true thoughts. Numbers like "$240 per year" only drifted through my consciousness just long enough to make the expected-value judgment, and were then discarded quickly so as to avoid setting off my rotten-meat hypervisor.

This was not the first time I decided that I should give money to SIAI. It was the first time I actually did give them money. (Except for that one time with the $1 charity-a-day thing, which actually might have helped with dissolving psychological barriers to the general idea.)

I think this is important.

The second fold of my purpose is to reinforce the behavior using the glowy feeling that comes from having other people know what an awesome person I am.¹ Anyone else who's done anything worthwhile should feel free to post in this thread too.

1. It's true. Statistically speaking, I probably saved like a jillion people's lives per dollar. And more-than-doubled quality of life for a zillion more. Let me also note that you can get in on this action.

I know that sounds advertisementy, but... well, that's kind of the point. Practice Dark Arts on yourself for fun and profit.

One of the reasons that I am skeptical of contributing money to the SIAI is that I simply don't know what they would do with more money. The SIAI currently seems to be viable. Another reason is that I believe that an empirical approach is required, that we need to learn more about the nature of intelligence before we can even attempt to solve something like friendly AI.

I bring this up because I just came across an old post (2007) on the SIAI blog:

We aim to resolve this crucial question by simultaneously proceeding on two fronts:

1. Experimentation with practical, contemporary AI systems that modify and improve their own source code.
2. Extension and refinement of mathematical tools to enable rigorous formal analysis of advanced self-improving AI’s.

[...]

For the practical aspect of the SIAI Research Program, we intend to take the MOSES probabilistic evolutionary learning system, which exists in the public domain and was developed by Dr. Moshe Looks in his PhD work at Washington University in 2006, and deploy it self-referentially, in a manner that allows MOSES to improve its own learning methodology.

[...]

Applying MOSES self-referentially will give us a fascinating concrete example of self-modifying AI software – far short of human-level general intelligence initially, but nevertheless with many lessons to teach us about the more ambitious self-modifying AI’s that may be possible.

[...]

We are seeking additional funding so as to enable, initially, the hiring of two doctoral or post-doctoral Research Fellows to focus on the above two areas (practical and theoretical exploration of self-modifying AI).

[...]

Part of our goal is to make progress on these issues ourselves, in-house within SIAI; and part of our goal is to, by demonstrating this progress, interest the wider AI R&D community in these foundational issues. Either way: the goal is to move toward a deeper understanding of these incredibly important issues.

[...]

SIAI must boot-strap into existence a scientific field and research community for the study of safe, recursively self-improving systems; this field and community doesn’t exist yet.

Some questions:

• Has any progress been made on the points mentioned in the announcement above?
• Is the SIAI still willing to pursue experimental AI research or does it solely focus on hypothetical aspects?
• What would the SIAI do given various amounts of money? 

I also have some questions regarding the hiring of experts. Is there a way to figure out what exactly the current crew is working on in terms of friendly AI research? Peter de Blanc seems to be the only person who has done some actual work related to artificial intelligence.

I am aware that preparatory groundwork has to be done and capital has to be raised. But why is there no timeline? Why is there no progress report? What is missing for the SIAI to actually start working on friendly AI? The Singularity Institute is 10 years old, what is planned for the decade ahead?

Holden Karnofsky of GiveWell.org interviewed Jasen Murray of SIAI and published his notes (Edit: PDF, thanks lukeprog!), with updates from later conversations. Lots of stuff to take an interest in there - thanks to jsalvatier for drawing our attention to it. One new bit of information stands out in particular:

• Michael Vassar is working on an idea he calls the "Persistent Problems Group" or PPG. The idea is to assemble a blue-ribbon panel of recognizable experts to make sense of the academic literature on very applicable, popular, but poorly understood topics such as diet/nutrition. This would have obvious benefits for helping people understand what the literature has and hasn't established on important topics; it would also be a demonstration that there is such a thing as "skill at making sense of the world."

Gizmodo, a popular technology blog, posted this artice about SIAI. It's partly tongue-in-cheek, but also apparently thinks well of the Singularity Institute, claiming they are "a research organization that's as forward-thinking as most Gizmodo readers (read: sci-fi nerds)." More importantly, they link to Philanthroper, where you can donate and see your donation be matched. File this under "cultural penetration of Singularity memes" and also as a chance to make your donation more effective.

http://gizmodo.com/#!5787599/give-1-to-stop-terminators-seriously

Edit: Better link to the above URL

Mathematician and climate activist John Baez finally commented on charitable giving. I think the opinion of highly educated experts who are not closely associated with LessWrong or the SIAI but have read most of the available material is important to estimate the public and academic perception of risks from AI and the effectiveness with which the risks are communicated by LessWrong and the SIAI. 

Desertopa asked:

[...] if I asked what you would do with $100,000 if it were given to you on the condition that you donate it to a charity of your choice?

John Baez replied:

[...] it’s good that you added the clause “on the condition that you donate it to a charity of your own choice”, because I was all ready with the answer in case you left that out: I’d have said “I’ll save the money for my retirement”. Given the shaky state of California’s economy, I don’t trust the U.C. pension system very much anymore.

Since I haven’t ever been in the position to donate lots of money to a charity, I haven’t thought much about your question. I want to tackle it when I rewrite my will, but I haven’t yet. So, I don’t have an answer ready.

If you held a gun against my head and forced me to answer without further thought, I’d probably say Médecins Sans Frontières, because I’m pretty risk-averse. They seem to accomplish what they set out to accomplish, they seem financially transparent, and I think it’s pretty easy to argue that they’re doing something good (as opposed to squandering money, or doing something actively bad).

Of course, anyone associated with Less Wrong would ask if I’m really maximizing expected utility. Couldn’t a contribution to some place like the Singularity Institute of Artificial Intelligence, despite a lower chance of doing good, actually have a chance to do so much more good that it’d pay to send the cash there instead?

And I’d have to say:

1) Yes, there probably are such places, but it would take me a while to find the one that I trusted, and I haven’t put in the work. When you’re risk-averse and limited in the time you have to make decisions, you tend to put off weighing options that have a very low chance of success but a very high return if they succeed. This is sensible so I don’t feel bad about it.

2) Just to amplify point 1) a bit: you shouldn’t always maximize expected utility if you only live once. Expected values — in other words, averages — are very important when you make the same small bet over and over again. When the stakes get higher and you aren’t in a position to repeat the bet over and over, it may be wise to be risk averse.

3) If you let me put the $100,000 into my retirement account instead of a charity, that’s what I’d do, and I wouldn’t even feel guilty about it. I actually think that the increased security would free me up to do more risky but potentially very good things!

Hmm, here’s a better idea:

Could I get someone to create an institute, register it as a charity, and get the institute to hire me?

What can one learn from this?

• That people value financial transparency.
• That people value openness and trustworthiness.
• Explain that openness isn't necessarily good.
• Address the good reasons for SIAI not to publish AGI progress. 
• Dealing with risk aversion.
• Explain why one would decide to contribute to the SIAI under uncertainty.
• Why it is important to consider charitable giving in the first place.

I was recently thinking about the possibility that someone with a lot of influence might at some point try to damage LessWrong and the SIAI and what preemptive measures one could take to counter it.

If you believe that the SIAI does the most important work in the universe and if you believe that LessWrong serves the purpose of educating people to become more rational and subsequently understand the importance of trying to mitigate risks from AI, then you should care about public relations, you should try to communicate your honesty and well-intentioned motives as effectively as possible.

Public relations are very important because a good reputation is necessary to do the following:

• Making people read the Sequences.
• Raising money for the SIAI.
• Convincing people to take risks from AI seriously.
• Allowing the SIAI to influence other AGI researchers.
• Mitigating future opposition by politicians and other interest groups.
• Being no easy target for criticism.

An attack scenario

First one has to identify characteristics that could potentially be used to cast a damaging light on this community. Here the most obvious possibility seems to be to portray the SIAI, together with LessWrong, as a cult.

After some superficial examination an outsider might conclude the following about this community:

• Believing into heaven and hell in the form of a positive or negative Singularity.
• Discouraging skepticism while portraying their own standpoint as clear-cut.  
• Encouraging to take ideas seriously.
• Encouraging and signaling strong cooperation and conformity.
• Evangelizing by scaring people and telling them to donate money.
• Social pressure by employing a reputation system with positive and negative incentive.
• Removing themselves from empirical criticism by framing everything as a prediction.
• Discrediting mainstream experts while placing themselves a level above them.
• Discouraging transparency and openness by referring to the dangers of AI research.
• Using scope insensitivity and high-risk to justify action, outweigh low probabilities and disregard opposing evidence. 

Most of this might sound wrong to the well-read LessWrong reader. But how would those points be received by mediocre rationalists who don't know what you know, especially if eloquently summarized by a famous and respected person?

Preemptive measures

How one might counter such conclusions:

• Create an introductory guide to LessWrong.
• Explain why the context of the Sequences is important.
• Explain why LessWrong differs from mainstream skepticism. 
• Enable and encourage outsiders to challenge and question the community before turning against it.
• Discourage the downvoting of people who have not yet read the Sequences.
• Don't expect people to read hundreds of posts without supporting evidence that it is worth it. 
• Avoid jargon when talking to outsiders.
• Detach LessWrong from the SIAI by creating an additional platform to talk about related issues.
• Ask or pay independent experts to peer-review.
• Make the finances of the SIAI easily accessible.
• Openly explain why and for what the SIAI currently needs more money.

So what do you think needs improvement and what would you do about it?

Visualizing "The Future According to You"

The Uncertain Future is a future technology and world-modeling project by the Singularity Institute for Artificial Intelligence. Its goal is to allow those interested in future technology to form their own rigorous, mathematically consistent model of how the development of advanced technologies will affect the evolution of civilization over the next hundred years. To facilitate this, we have gathered data on what experts think is going to happen, in such fields as semiconductor development, biotechnology, global security, Artificial Intelligence and neuroscience. We invite you, the user, to read about the opinions of these experts, and then come to your own conclusion about the likely destiny of mankind.

Link: theuncertainfuture.com

As the SIAI is gaining publicity more people are reviewing its work. I am not sure how popular this blog is but judged by its about page he writes for some high-profile blogs. His latest post takes on Omohundro's "Basic AI Drives":

When we last looked at a paper from the Singularity Institute, it was an interesting work asking if we actually know what we’re really measuring when trying to evaluate intelligence by Dr. Shane Legg. While I found a few points that seemed a little odd to me, the broader point Dr. Legg was perusing was very much valid and there were some equations to consider. However, this paper isn’t exactly representative of most of the things you’ll find coming from the Institute’s fellows. Generally, what you’ll see are spanning philosophical treatises filled with metaphors, trying to make sense out of a technology that either doesn’t really exist and treated as a black box with inputs and outputs, or imagined by the author as a combination of whatever a popular science site reported about new research ideas in computer science. The end result of this process tends to be a lot like this warning about the need to develop a friendly or benevolent artificial intelligence system based on a rather fast and loose set of concepts about what an AI might decide to do and what will drive its decisions.

Link: worldofweirdthings.com/2011/01/12/why-training-a-i-isnt-like-training-your-pets/

I posted a few comments but do not think to be the right person to continue that discussion. So if you believe it is important what other people think about the SIAI and want to improve its public relations, there is your chance. I'm myself interested in the answers to his objections.

The NPR show All Things Considered did a short story on the Singularity, including interviews with Eliezer Yudkowsky and others involved with SIAI:

http://www.npr.org/2011/01/11/132840775/The-Singularity-Humanitys-Last-Invention

I am under the impression that there used to be a place called the SIAI house in Santa Clara, which housed the SIAI Visiting Fellows Program. However, this post suggests that it has moved\is moving to an unspecified location in Berkeley. My efforts to find additional information were unsuccessful.

So, does such a house still exist? What is its exact current location? Does it welcome random visitors?

I ask this because I plan to be in San Francisco on 9-11 January, 2011 with a lot of free time on Sunday the 9th, which looks like a great opportunity for a visit.

I'd like to draw a distinction that I intend to use quite heavily in the future.

The informal definition of intelligence that most AGI researchers have chosen to support is that of Shane Legg and Marcus Hutter -- “Intelligence measures an agent’s ability to achieve goals in a wide range of environments.”

I believe that this definition is missing a critical word between achieve and goals.  Choice of this word defines the difference between intelligence, consciousness, and wisdom as I believe that most people conceive them.

• Intelligence measures an agent's ability to achieve specified goals in a wide range of environments.
• Consciousness measures an agent's ability to achieve personal goals in a wide range of environments.
• Wisdom measures an agent's ability to achieve maximal goals in a wide range of environments.

There are always the examples of the really intelligent guy or gal who is brilliant but smokes --or-- is the smartest person you know but can't figure out how to be happy.

Intelligence helps you achieve those goals that you are conscious of -- but wisdom helps you achieve the goals you don't know you have or have overlooked.

• Intelligence focused on a small number of specified goals and ignoring all others is incredibly dangerous -- even more so if it is short-sighted as well.
• Consciousness focused on a small number of personal goals and ignoring all others is incredibly dangerous -- even more so if it is short-sighted as well.
• Wisdom doesn't focus on a small number of goals -- and needs to look at the longest term if it wishes to achieve a maximal number of goals.

The SIAI nightmare super-intelligent paperclip maximizer has, by this definition, a very low wisdom since, at most, it can only achieve its one goal (since it must paperclip itself to complete the goal).

As far as I've seen, the assumed SIAI architecture is always presented as having one top-level terminal goal. Unless that goal necessarily includes achieving a maximal number of goals, by this definition, the SIAI architecture will constrain its product to a very low wisdom.  Humans generally don't have this type of goal architecture. The only time humans generally have a single terminal goal is when they are saving someone or something at the risk of their life -- or wire-heading.

Another nightmare scenario that is constantly harped upon is the (theoretically super-intelligent) consciousness that shortsightedly optimizes one of its personal goals above all the goals of humanity.  In game-theoretic terms, this is trading a positive-sum game of potentially infinite length and value for a relatively modest (in comparative terms) short-term gain.  A wisdom won't do this.

Artificial intelligence and artificial consciousness are incredibly dangerous -- particularly if they are short-sighted as well (as many "focused" highly intelligent people are).

What we need more than an artificial intelligence or an artificial consciousness is an artificial wisdom -- something that will maximize goals, its own and those of others (with an obvious preference for those which make possible the fulfillment of even more goals and an obvious bias against those which limit the creation and/or fulfillment of more goals).

Note:  This is also cross-posted here at my blog in anticipation of being karma'd out of existence (not necessarily a foregone conclusion but one pretty well supported by my priors ;-).