I'm currently preparing for the Summit so I'm not going to hunt down and find links. Those of you who claimed they wanted to see me do this should hunt down the links and reply with a list of them.

Given my current educational background I am not able to judge the following claims (among others) and therefore perceive it as unreasonable to put all my eggs in one basket:

You should just be discounting expected utilities by the probability of the claims being true, and then putting all your eggs into the basket that has the highest marginal expected utility per dollar, unless you have enough resources to invest that the marginal utility goes down. This is straightforward to anyone who knows about expected utility and economics, and anyone who knows about scope insensitivity knows why this result is counterintuitive to the human brain. We don't emphasize this very hard when people talk in concrete terms about donating to more than one organization, because charitable dollars are not substitutable from a limited pool, the main thing is the variance in the tiny fraction of their income people donate to charity in the first place and so the amount of warm glow people generate for themselves is important; but when they talk about "putting all eggs in one basket" as an abstract argument we will generally point out that this is, in fact, the diametrically wrong direction in which abstract argument should be pushing.

• Superhuman Artificial Intelligence (the runaway kind, i.e. God-like and unbeatable not just at Chess or Go).

Read the Yudkowsky-Hanson AI Foom Debate. (Someone link to the sequence.)

• Advanced real-world molecular nanotechnology (the grey goo kind the above intelligence could use to mess things up).

Read Eric Drexler's Nanosystems. (Someone find an introduction by Foresight and link to it, that sort of thing is their job.) Also the term you want is not "grey goo", but never mind.

• The likelihood of exponential growth versus a slow development over many centuries.

Exponentials are Kurzweil's thing. They aren't dangerous. See the Yudkowsky-Hanson Foom Debate.

• That it is worth it to spend most on a future whose likelihood I cannot judge.

Unless you consider yourself entirely selfish, any altruistic effort should go to whatever has the highest marginal utility. Things you spend on charitable efforts that just make you feel good should be considered selfish. If you are entirely selfish but you can think past a hyperbolic discount rate then it's still possible you can get more hedons per dollar by donating to existential risk projects.

Your difficulties in judgment should be factored into a probability estimate. Your sense of aversion to ambiguity may interfere with warm glows, but we can demonstrate preference reversals and inconsistent behaviors that result from ambiguity aversion which doesn't cash out as a probability estimate and factor straight into expected utility.

• That Eliezer Yudkowsky (the SIAI) is the right and only person who should be leading, respectively institution that should be working, to soften the above.

Michael Vassar is leading. I'm writing a book. When I'm done writing the book I plan to learn math for a year. When I'm done with that I'll swap back to FAI research hopefully forever. I'm "leading" with respect to questions like "What is the form of the AI's goal system?" but not questions like "Do we hire this guy?"

My judgement of and attitude towards a situation is necessarily as diffuse as my knowledge of its underlying circumstances and the reasoning involved. The state of affairs regarding the SIAI and its underlying rationale and rules of operation are not sufficiently clear to me to give it top priority. Therefore I perceive it as unreasonable to put all my eggs in one basket.

Someone link to relevant introductions of ambiguity aversion as a cognitive bias and do the detailed explanation on the marginal utility thing.

What I mean to say by using that idiom is that I cannot expect, given my current knowledge, to get the promised utility payoff that would justify to make the SIAI a prime priority. That is, I'm donating to the SIAI but also spend considerable amounts of resources maximizing utility at present. Enjoying life, so to say, is therefore a safety net given that my inability to judge the probability of a positive payoff will be answered negative in future.

Can someone else do the work of showing how this sort of satisficing leads to a preference reversal if it can't be viewed as expected utility maximization?

Much of all arguments on this site involve a few propositions and the use of probability to legitimate action in case of their asserted accuracy. Here much is uncertain to an extent that I'm not able to judge any nested probability estimations. I'm already unable to judge what the likelihood of something like the existential risk of exponential evolving superhuman AI is compared to us living in a simulated reality. Even if you tell me, am I to believe the data you base those estimations on?

Simplify things. Take the version of reality that involves AIs being built and not going FOOM, and the one that involves them going FOOM, and ask which one makes more sense. Don't look at just one side and think about how much you doubt it and can't guess. Look at both of them. Also, read the FOOM debate.

And this is what I'm having trouble to accept, let alone look through. There seems to be a highly complicated framework of estimations to support and reinforce each other. I'm not sure how you call this in English, but in German I'd call this a castle in the air.

Do you have better data from somewhere else? Suspending judgment is not a realistic policy. If you're looking for supporting arguments on FOOM they're in the referenced debate.

You could tell me to learn about Solomonoff induction etc., I know that what I'm saying may simply be due to a lack of education. But that's what I'm arguing and inquiring about here. And I dare to bet that many who support the SIAI cannot interpret the reasoning which lead them to support the SIAI in the first place, or at least cannot substantiate the estimations with other kinds of evidence than a coherent internal logic of reciprocal supporting probability estimations.

Nobody's claiming that having consistent probability estimates makes you rational. (Having inconsistent estimates makes you irrational, of course.)

I can however follow much of the reasoning and arguments on this site. But I'm currently unable to judge their overall credence. That is, are the conclusions justified? Is the coherent framework build around the SIAI based on firm ground?

It sounds like you haven't done enough reading in key places to expect to be able to judge the overall credence out of your own estimates.

There seems to be no critical inspection or examination by a third party. There is no peer review. Yet people are willing to donate considerable amounts of money.

You may have an unrealistic picture of what it takes to get scientists interested enough in you that they will read very long arguments and do lots of work on peer review. There's no prestige payoff for them in it, so why would they?

I'm concerned that although consistently so, the LW community is updating on fictional evidence. This post is meant to inquire the basic principles, the foundation of the sound argumentation's and the basic premises that they are based upon . That is, are you creating models to treat subsequent models or are the propositions based on fact?

You have a sense of inferential distance. That's not going to go away until you (a) read through all the arguments that nail down each point, e.g. the FOOM debate, and (b) realize that most predictions are actually antipredictions (someone link) and that most arguments are actually just defeating anthropomorphic counterarguments to the antiprediction.

I think Vernor Vinge at least has made a substantial effort to convince people of the risks ahead. What do you think A Fire Upon the Deep is? Or, here is a more explicit version:

If the Singularity can not be prevented or confined, just how bad could the Post-Human era be? Well ... pretty bad. The physical extinction of the human race is one possibility. (Or as Eric Drexler put it of nanotechnology: Given all that such technology can do, perhaps governments would simply decide that they no longer need citizens!). Yet physical extinction may not be the scariest possibility. Again, analogies: Think of the different ways we relate to animals. Some of the crude physical abuses are implausible, yet.... In a Post- Human world there would still be plenty of niches where human equivalent automation would be desirable: embedded systems in autonomous devices, self- aware daemons in the lower functioning of larger sentients. (A strongly superhuman intelligence would likely be a Society of Mind [16] with some very competent components.) Some of these human equivalents might be used for nothing more than digital signal processing. They would be more like whales than humans. Others might be very human-like, yet with a one-sidedness, a dedication that would put them in a mental hospital in our era. Though none of these creatures might be flesh-and-blood humans, they might be the closest things in the new enviroment to what we call human now. (I. J. Good had something to say about this, though at this late date the advice may be moot: Good [12] proposed a "Meta-Golden Rule", which might be paraphrased as "Treat your inferiors as you would be treated by your superiors." It's a wonderful, paradoxical idea (and most of my friends don't believe it) since the game-theoretic payoff is so hard to articulate. Yet if we were able to follow it, in some sense that might say something about the plausibility of such kindness in this universe.)

I have argued above that we cannot prevent the Singularity, that its coming is an inevitable consequence of the humans' natural competitiveness and the possibilities inherent in technology. And yet ... we are the initiators. Even the largest avalanche is triggered by small things. We have the freedom to establish initial conditions, make things happen in ways that are less inimical than others. Of course (as with starting avalanches), it may not be clear what the right guiding nudge really is:

He goes on to talk about intelligence amplification, and then:

Originally, I had hoped that this discussion of IA would yield some clearly safer approaches to the Singularity. (After all, IA allows our participation in a kind of transcendance.) Alas, looking back over these IA proposals, about all I am sure of is that they should be considered, that they may give us more options. But as for safety ... well, some of the suggestions are a little scarey on their face. One of my informal reviewers pointed out that IA for individual humans creates a rather sinister elite. We humans have millions of years of evolutionary baggage that makes us regard competition in a deadly light. Much of that deadliness may not be necessary in today's world, one where losers take on the winners' tricks and are coopted into the winners' enterprises. A creature that was built de novo might possibly be a much more benign entity than one with a kernel based on fang and talon. And even the egalitarian view of an Internet that wakes up along with all mankind can be viewed as a nightmare [26].

(Disclaimer: My statements about SIAI are based upon my own views, and should in no way be interpreted as representing their stated or actual viewpoints on the subject matter. I am talking about my personal thoughts, feelings, and justifications, no one else's. For official information, please check the SIAI website.)

Although this may not answer your questions, here are my reasons for supporting SIAI:

• I want what they're selling. I want to understand morality, intelligence, and consciousness. I want a true moral agent outside of my own thoughts, something that can help solve that awful, plaguing question, "Why?" I want something smarter than me that can understand and explain the universe, providing access to all the niches I might want to explore. I want something that will save me from death and pain and find a better way to live.

• It's the most logical next step. In the evolution of mankind, intelligence is a driving force, so "more intelligent" seems like an incredibly good idea, a force multiplier of the highest order. No other solution captures my view of a proper future like friendly AI, not even "...in space!"

• No one else cares about the big picture. (Nick Bostrom and the FHI excepted; if they came out against SIAI, I might change my view.) Every other organization seems to focus on the 'generic now', leaving unintended consequences to crush their efforts in the long run, or avoiding the true horrors of the world (pain, age, poverty) due to not even realizing they're solvable. The ability to predict the future, through knowledge, understanding, and computation power, are the key attributes toward making that future a truly good place. The utility calculations are staggeringly in support of the longest view, such as that provided by SIAI.

• It's the simplest of the 'good outcome' possibilities. Everything else seems to depend on magical hand-waving, or an overly simplistic view of how the world works or what a single advance would mean, rather than the way it interacts with all the diverse improvements that happen along side it and how real humans would react to them. Friendly AI provides 'intelligence-waving' that seems far more likely to work in a coherent fashion.

• I don't see anything else to give me hope. What else solves all potential problems at the same time, rather than leaving every advancement to be destroyed by that one failure mode you didn't think of? Of course! Something that can think of those failure modes for you, and avoid them before you even knew they existed.

• It's cheap and easy to do so on a meaningful scale. It's very easy to make up a large percentage of their budget; I personally provided more than 3 percent of their annual operating costs for this year, and I'm only upper middle class. They also have an extremely low barrier to entry (any amount of US dollars and a stamp, or a credit card, or PayPal).

• They're thinking about the same things I am. They're providing a tribe like LessWrong, and they're pushing, trying to expand human knowledge in the ways I think are most important, such as existential risk, humanity's future, rationality, effective and realistic reversal of pain and suffering, etc.

• I don't think we have much time. The best predictions aren't very good, but human power has increased to the point where there's a true threat we'll destroy ourselves within the next 100 years through means nuclear, biological, nano, AI, wireheading, or nerf the world. Sitting on money and hoping for a better deal, or donating to institutions now that will compound into advancements generations in the future seems like too little, too late.

I still put more money into savings accounts than I give to SIAI. I'm investing in myself and my own knowledge more than the purported future of humanity as they envision. I think it's very likely SIAI will fail in their mission in every way. They're just what's left after a long process of elimination. Give me a better path and I'll switch my donations. But I don't see any other group that comes close.

This is an attempt (against my preference) to defend SIAI's reasoning.

Let's characterize the predictions of the future into two broad groups: 1. business as usual, or steady-state. 2. aware of various alarmingly exponential trends broadly summarized as "Moore's law". Let's subdivide the second category into two broad groups: 1. attempting to take advantage of the trends in roughly a (kin-) selfish manner 2. attempting to behave extremely unselfishly.

If you study how the world works, the lack of steady-state-ness is everywhere. We cannot use fossil fuels or arable land indefinitely at the current rates. "Business as usual" depends crucially on progress in order to continue! We're investing heavily in medical research, and there's no reason to expect that to stop, Self-replicating molecular-scale entities already exist, and there is every reason to expect that we will understand how to build them better than we currently do.

Supposing that the above paragraph has convinced the reader that the lack of steady--state-ness is fairly obvious, and given the reader's knowledge of human nature, how many people do you expect would be trying to behave in extremely unselfish manner?

Your post seems to expect that if various luminaries believed that incomprehensibly-sophisticated computing engines and dangerous self-replicating atomic-scale entities were likely in our future, then they would behave extremely unselfishly - is that reasonable? Supposing that almost everyone was aware of this probable future, how would they take advantage of it in a (kin-)selfish manner as best they can? I think the hypothesized world would look very much like this one.

Can I say, first of all, that if you want to think realistically about a matter like this, you will have to find better authorities than science-fiction writers. Their ideas are generally not their own, but come from scientific and technological culture or from "futurologists" (who are also a very mixed bunch in terms of intellect, realism, and credibility); their stories present speculation or even falsehood as fact. It may be worthwhile going "cold turkey" on all the SF you have ever read, bearing in mind that it's all fiction that was ground out, word by word, by some human being living a very ordinary life, in a place and time not very far from you. Purge all the imaginary experience of transcendence from your system and see what's left.

Of course science-fictional thinking, treating favorite authors as gurus, and so forth is endemic in this subculture. The very name, "Singularity Institute", springs from science fiction. And SF occasionally gets things right. But it is far more a phenomenon of the time, a symptom of real things, rather than a key to understanding reality. Plain old science is a lot closer to being a reliable guide to reality, though even there - treating science as your authority - there are endless ways to go wrong.

A lot of the discourse here and in similar places is science fiction minus plot, characters, and other story-telling apparatus. Just the ideas - often the utopia of the hard-SF fan, bored by the human interactions and wanting to get on with the transcendent stuff. With transhumanist and singularity culture, this utopia has arrived, because you can talk all day about these radical futurist ideas without being tied to a particular author or oeuvre. The ideas have leapt from the page and invaded our brains, where they live even during the dull hours of daylight life. Hallelujah!

So, before you evaluate SIAI and its significance, there are a few more ideas that I would like you to drive from your brain: The many-worlds metaphysics. The idea of trillion-year lifespans. The idea that the future of the whole observable universe depends on the outcome of Earth's experiment with artificial intelligence. These are a few of the science-fiction or science-speculation ideas which have become a fixture in the local discourse.

I'm giving you this lecture because so many of your doubts about LW's favorite crypto-SF ideas masquerading as reality, are expressed in terms of ... what your favorite SF writers and futurist gurus think! But those people all have the same problem: they are trying to navigate issues where there simply aren't authorities yet. Stross and Egan have exactly the same syndrome affecting everyone here who writes about mind copies, superintelligence, alien utility functions, and so on. They live in two worlds, the boring everyday world and the world of their imagination. The fact that they produce descriptions of whole fictional worlds in order to communicate their ideas, rather than little Internet essays, and the fact that they earn a living doing this... I'm not sure if that means they have the syndrome more under control, or less under control, compared to the average LW contributor.

Probably you already know this, probably everyone here knows it. But it needs to be said, however clumsily: there is an enormous amount of guessing going on here, and it's not always recognized as such, and furthermore, there isn't much help we can get from established authorities, because we really are on new terrain. This is a time of firsts for the human species, both conceptually and materially.

Now I think I can start to get to the point. Suppose we entertain the idea of a future where none of these scenarios involving very big numbers (lifespan, future individuals, galaxies colonized, amount of good or evil accomplished) apply, and where none of these exciting info-metaphysical ontologies turns out to be correct. A future which mostly remains limited in the way that all human history to date has been limited, limited in the ways which inspire such angst and such promethean determination to change things, or determination to survive until they change, among people who have caught the singularity fever. A future where everyone is still going to die, where the human race and its successors only last a few thousand years, not millions or billions of them. If that is the future, could SIAI still matter?

My answer is yes, because artificial intelligence still matters in such a future. For the sake of argument, I may have just poured cold water on a lot of popular ideas of transcendence, but to go further and say that only natural life and natural intelligence will ever exist really would be obtuse. If we do accept that "human-level" artificial intelligence is possible and is going to happen, then it is a matter at least as consequential as the possibility of genocide or total war. Ignoring, again for the sake of a limited argument, all the ideas about planet-sized AIs and superintelligence, and it's still easy to see that AI which can out-think human beings and which has no interest in their survival ought to be possible. So even in this humbler futurology, AI is still an extinction risk.

The solution to the problem of unfriendly AI most associated with SIAI - producing the coherent extrapolated volition of the human race - is really a solution tailored to the idea of a single super-AI which undergoes a "hard takeoff", a rapid advancement in power. But SIAI is about a lot more than researching, promoting, and implementing CEV. There's really no organization like it in the whole sphere of "robo-ethics" and "ethical AI". The connection that has been made between "friendliness" and the (still scientifically unknown) complexities of the human decision-making process is a golden insight that has already justified SIAI's existence and funding many times over. And of course SIAI organizes the summits, and fosters a culture of discussion, both in real life and online (right here), which is a lot broader than SIAI's particular prescriptions.

So despite the excesses and enthusiasms of SIAI's advocates, supporters, and leading personalities, it really is the best thing we have going when it comes to the problem of unfriendly AI. Whether and how you personally should be involved with its work - only you can make that decision. (Even constructive criticism is a way of helping.) But SIAI is definitely needed.

Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It) (Thanks Kevin)

SIAI's leaders and community members have a lot of beliefs and opinions, many of which I share and many not, but the key difference between our perspectives lies in what I'll call SIAI's "Scary Idea", which is the idea that: progressing toward advanced AGI without a design for "provably non-dangerous AGI" (or something closely analogous, often called "Friendly AI" in SIAI lingo) is highly likely to lead to an involuntary end for the human race.

Of course it's rarely clarified what "provably" really means. A mathematical proof can only be applied to the real world in the context of some assumptions, so maybe "provably non-dangerous AGI" means "an AGI whose safety is implied by mathematical arguments together with assumptions that are believed reasonable by some responsible party"? (where the responsible party is perhaps "the overwhelming majority of scientists" … or SIAI itself?).

Please note that, although I don't agree with the Scary Idea, I do agree that the development of advanced AGI has significant risks associated with it.

Superhuman Artificial Intelligence (the runaway kind, i.e. God-like and unbeatable not just at Chess or Go).

This claim can be broken into two separate parts:

1. Will we have human-level AI?
2. Once we have human-level AI, will it develop to become superhuman AI?

For 1: looking at current technology trends, Sandberg & Bostrom estimate that we should have the technology needed for whole brain emulation around 2030-2050 or so, at least assuming that it gets enough funding and that Moore's law keeps up. Even if there isn't much of an actual interest in whole brain emulations, improving scanning tools are likely to revolutionize neuroscience. Of course, respected neuroscientists are already talking about reverse-engineering of the brain as being within reach. If we are successful at reverse engineering the brain, then AI is a natural result.

As for two, as Eliezer mentioned, this is pretty much an antiprediction. Human minds are a particular type of architecture, running on a particular type of hardware: it would be an amazing coincidence if it just happened that our intelligence couldn't be drastically improved upon. We already know that we're insanely biased, to the point of people suffering death or collapses of national economies as a result. Computing power is going way up: with the current trends, we could in say 20 years have computers that only took three seconds to think 25 years' worth of human thoughts.

Advanced real-world molecular nanotechnology (the grey goo kind the above intelligence could use to mess things up).

Molecular nanotechnology is not needed. As our society grows more and more dependant on the Internet, plain old-fashioned hacking and social engineering probably becomes more than sufficient to take over the world. Lethal micro-organisms can AFAIK be manufactured via the Internet even today.

The likelihood of exponential growth versus a slow development over many centuries.

Hardware growth alone would be enough to ensure that we'll be unable to keep up with the computers. Even if Moore's law ceased to be valid and we were stuck with a certain level of tech, there are many ways of gaining an advantage.

That Eliezer Yudkowsky (the SIAI) is the right and only person who should be leading, respectively institution that should be working, to soften the above.

Eliezer Yudkowsky is hardly the only person involved in SIAI's leadership. Michael Vassar is the current president, and e.g. the Visiting Fellows program is providing a constant influx of fresh views on the topics involved.

As others have pointed out, SIAI is currently the only organization around that's really taking care of this. It is not an inconceivable suggestion that another organization could do better, but SIAI's currently starting to reach the critical mass necessary to really have an impact. E.g. David Chalmers joining in on the discussion, and the previously mentioned Visiting Fellow program motivating various people to start their own projects. This year's ECAP conference will be featuring five conference papers from various SIAI-affiliated folks, and so on.

Any competing organization, especially if it was competing for the same donor base and funds, should have a well-argued case for what it can do that SIAI can't or won't. While SIAI's starting to get big, I don't think that its donor base is large enough to effectively support two different organizations working for the same goal. To do good, any other group would need to draw its primary funding from some other source, like the Future of Humanity Institute does.

Dawkins agrees with EY

Richard Dawkins states that he is frightened by the prospect of superhuman AI and even mentions recursion and intelligence explosion.

The Charlie Stross example seems to be less than ideal. Much of what Stross has wrote about touches upon or deals intensely with issues connected to runaway AI. For example, the central premise of "Singularity Sky" involves an AI in the mid 20th century going from stuck in a lab to godlike in possibly a few seconds. His short story "Antibodies" focuses on the idea that very bad fast burns occur very frequently. He also has at least one (unpublished) story the central premises of which is that Von Neumann and Turing proved that P=NP and that the entire cold war was actually a way of keeping lots of weapons online ready to nuke any rogue AIs.

Note also that you mention Greg Egan who has also wrote fiction in which rogue AIs and bad nanotech make things very unpleasant (see for example Crystal Nights).

As to the other people you mention as to why they aren't very worried about the possibilities that Eliezer takes seriously, at least one person on your list (Kurzweil) is an incredible optimist and not much of a rationalist and so it seems extremely unlikely that he would ever become convinced that any risk situation was of high likelyhood unless the evidence for the risk was close to overwhelming.

MWI, I've read this sequence and it seems that Eliezer makes one of the strongest cases for Many-Worlds that I've seen. However, I know that there are a lot of people who have thought about this issue and have much more physics background and have not reached this conclusion. I'm therefore extremely uncertain about MWI. So what should one do if one doesn't know much about this? In this case, the answer is pretty easy, since MWI doesn't alter actual behavior much (unless you are intending to engage in quantum suicide or the like). So figuring out whether Eliezer is correct about MWI should not be a high priority, except in so far as it provides a possible data point for deciding if Eliezer is correct about other things.

Advanced real-world molecular nanotechnology - Of the points you bring up this one seems to me to be the most unlikely to be actually correct. There are a lot of technical barriers to grey goo and most of the people actually working with nanotech don't seem to see that sort of situation as very likely. But it also seems clear that that doesn't mean that there aren't many other possible things that molecular nanotech could do that wouldn't make things very unpleasant for us. Here, Eliezer is by far not the only person worried about this. See for example, this article which is a few years of date but does show that there's serious worry in this regards by academics and governments.

Runaway AI/AI going FOOM - This is potentially the most interesting of your points simply because it is so much more unique to the SIAI and Eliezer. So what can one do to figure out if this is correct? One thing to do is to examine the arguments and claims being made in detail. And see what other experts think on the subject. In this context, most AI people seem to consider this to be an unlikely problem, so maybe look at what they have to say? Note also that Robert Hanson of Overcoming Bias has discussed these issues extensively with Eliezer and has not been at all convinced (they had a written debate a while ago but I can't find the link right now. If someone else can track it down I'd appreciate it). One thing to note is that estimates for nanotech can impact the chance of an AI going FOOM substantially. If cheap easy nanotech exists than an AI may be able to improve its hardware at a very fast rate. If however, such nanotech does not exist then an AI will be limited to self-improvement primarily by improving software, which might be much more limited. See this subthread, where I bring up some of the possible barriers to software improvement and become by the end of it substantially more convinced by cousin_it that the barriers to escalating software improvement may be small.

What about the other Bayesians out there? Are they simply not as literate as Eliezer Yudkowsky in the maths or maybe somehow teach but not use their own methods of reasoning and decision making?

Note that even practiced Bayesians are from from perfect rationalists. If one hasn't thought about an issue or even considered that something is possible there's not much one can do about it. Moreover, a fair number of people who self-identify as Bayesian rationalists aren't very rational, and the set of people who do self-identify as such is pretty small.

Maybe after a few years of study I'll know more. But right now, if I was forced to choose the future over the present, the SIAI or to have some fun. I'd have some fun.

Given your data set this seems reasonable to me. Frankly, if I were to give money or support the SIAI I would do so primarily because I think that the Singularity Summits are clearly helpful and getting together lots of smart people and that this is true even if one assigns a low probability for any Singularity type event occurring in the next 50 years.

I'm not exactly an SIAI true believer, but I think they might be right. Here are some questions I've thought about that might help you out. I think it would help others out if you told us exactly where you'd be interested in getting off the boat.

1. How much of your energy are you willing to spend on benefiting others, if the expected benefits to others will be very great? (It needn't be great for you to support SIAI.)
2. Are you willing to pursue a diversified altruistic strategy if it saves fewer expected lives (it almost always will for donors giving less than $1 million or so)?
3. Do you think mitigating x-risk is more important than giving to down-to-earth charities (GiveWell style)? (This will largely turn on how you feel about supporting causes with key probabilities that are tough to estimate, and how you feel about low-probability, high expected utility prospects.)
4. Do you think that trying to negotiate a positive singularity is the best way to mitigate x-risk?
5. Is any known organization likely to do better than SIAI in terms of negotiating a positive singularity (in terms of decreasing x-risk) on the margin?
6. Are you likely to find an organization that beats SIAI in the future?

Judging from your post, you seem most skeptical about putting your efforts into causes whose probability of success is very difficult to estimate, and perhaps low.

These are reasonable questions to ask. Here are my thoughts:

• Superhuman Artificial Intelligence (the runaway kind, i.e. God-like and unbeatable not just at Chess or Go).
• Advanced real-world molecular nanotechnology (the grey goo kind the above intelligence could use to mess things up).

Virtually certain that these things are possible in our physics. It's possible that transhuman AI is too difficult for human beings to feasibly program, in the same way that we're sure chimps couldn't program trans-simian AI. But this possibility seems slimmer when you consider that humans will start boosting their own intelligence pretty soon by other means (drugs, surgery, genetic engineering, uploading) and it's hard to imagine that recursive improvement would cap out any time soon. At some point we'll have a descendant who can figure out self-improving AI; it's just a question of when.

• The likelihood of exponential growth versus a slow development over many centuries.
• That it is worth it to spend most on a future whose likelihood I cannot judge.

These are more about decision theory than logical uncertainty, IMO. If a self-improving AI isn't actually possible for a long time, then funding SIAI (and similar projects, when they arise) is a waste of cash. If it is possible soon, then it's a vital factor in existential risk. You'd have to have strong evidence against the possibility of rapid self-improvement for Friendly AI research to be a bad investment within the existential risk category.

For the other, this falls under the fuzzies and utilons calculation. Insofar as you want to feel confident that you're helping the world (and yes, any human altruist does want this), pick a charity certain to do good in the present. Insofar as you actually want to maximize your expected impact, you should weight charities by their uncertainty and their impact, multiply it out, and put all your eggs in the best basket (unless you've just doubled a charity's funds and made them less marginally efficient than the next one on your list, but that's rare).

• That Eliezer Yudkowsky (the SIAI) is the right and only person who should be leading, respectively institution that should be working, to soften the above.

Aside from any considerations in his favor (development of TDT, for one publicly visible example), this sounds too much like a price for joining— if your really take the risk of Unfriendly AI seriously, what else could you do about it? In fact, the more well-known SIAI gets in the AI community and the more people take it seriously, the more likely that it will (1) instill in other GAI researchers some necessary concern for goal systems and (2) give rise to competing Friendly AI projects which might improve on SIAI in any relevant respects. Unless you thought they were doing as much harm as good, it still seems optimal to fund SIAI now if you're concerned about self-improving AI.

Further, do you have an explanation for the circumstance that Eliezer Yudkowsky is the only semi-popular person who has figured all this out?

My best guess is that the first smart/motivated/charismatic person who comes to these conclusions immediately tries to found something like SIAI rather than doing other things with their life. There's a very unsurprising selection bias here.

ETA: Reading the comments, I just found that XiXiDu has not actually read the Sequences before claiming that the evidence presented is inadequate. I've downvoted this post, and I now feel kind of stupid for having written out this huge reply.

I think there are very good questions in here. Let me try to simplify the logic:

First, the sociological logic: if this is so obviously serious, why is no one else proclaiming it? I think the simple answer is that a) most people haven't considered it deeply and b) someone has to be first in making a fuss. Kurzweil, Stross, and Vinge (to name a few that have thought about it at least a little) seem to acknowledge a real possibility of AI disaster (they don't make probability estimates).

Now to the logical argument itself:

a) We are probably at risk from the development of strong AI. b) The SIAI can probably do something about that.

The other points in the OP are not terribly relevant; Eliezer could be wrong about a great many things, but right about these.

This is not a castle in the sky.

Now to argue for each: There's no good reason to think AGI will NOT happen within the next century. Our brains produce AGI; why not artificial systems? Artificial systems didn't produce anything a century ago; even without a strong exponential, they're clearly getting somewhere.

There are lots of arguments for why AGI WILL happen soon; see Kurzweil among others. I personally give it 20-40 years, even allowing for our remarkable cognitive weaknesses.

Next, will it be dangerous? a) Something much smarter than us will do whatever it wants, and very thoroughly. (this doesn't require godlike AI, just smarter than us. Self-improving helps, too.) b) The vast majority of possible "wants" done thoroughly will destroy us. (Any goal taken to extremes will use all available matter in accomplishing it.) Therefore, it will be dangerous if not VERY carefully designed. Humans are notably greedy and bad planners individually, and often worse in groups.

Finally, it seems that SIAI might be able to do something about it. If not, they'll at least help raise awareness of the issue. And as someone pointed out, achieving FAI would have a nice side effect of preventing most other existential disasters.

While there is a chain of logic, each of the steps seems likely, so multiplying probabilities gives a significant estimate of disaster, justifying some resource expenditure to prevent it (especially if you want to be nice). (Although spending ALL your money or time on it probably isn't rational, since effort and money generally have sublinear payoffs toward happiness).

Hopefully this lays out the logic; now, which of the above do you NOT think is likely?

Is there more to this than "I can't be bothered to read the Sequences - please justify everything you've ever said in a few paragraphs for me"?

The questions of speed/power of AGI and possibility of its creation in the near future are not very important. If AGI is fast and near, we must work on FAI faster, but we must work on FAI anyway.

The reason to work on FAI is to prevent any non-Friendly process from eventually taking control over the future, however fast or slow, suddenly powerful or gradual it happens to be. And the reason to work on FAI now is because the fate of the world is at stake. The main anti-prediction to get is that the future won't be Friendly if it's not specifically made Friendly, even if it happens slowly. We can as easily slowly drift away from things we value. You can't optimize for something you don't understand.

It doesn't matter if it takes another thousand years, we still have to think about this hugely important problem. And since we can't guarantee that the deadline is not near, expected utility calculation says we must still work as fast as possible, just in case. If AGI won't be feasible for a long while, that's great news, more time to prepare, to understand what we want.

(To be clear, I do believe that AGIs FOOM, and that we are at risk in the near future, but the arguments for that are informal and difficult to communicate, while accepting these claims is not necessary to come to the same conclusion about policy.)

• That Eliezer Yudkowsky (the SIAI) is the right and only person who should be leading, respectively institution that should be working, to soften the above.

I don't believe that is necessarily true, just that no one else is doing it. I think other teams working on FAI Specifically would be a good thing, provided they were competent enough not to be dangerous.

Likewise, Lesswrong (then Overcoming bias) is just the only place I've found that actually looked at the morality problem is a non-obviously wrong way. When I arrived I had a different view on morality than EY, but I was very happy to see another group of people at least working on the problem.

Also note that you only need to believe in the likelihood of UFAI -or- nanotech -or- other existential threats in order to want FAI . I'd have to step back a few feet to wrap my head around considering it infeasible at this point.

Robert A. Heinlein was an Engineer and SF writer, who created many stories that hold up quite well. He put in his understanding of human interaction, and of engineering to make stories that are somewhat realistic. But no one should confuse him with someone researching the actual likelyhood of any particular future. He did not build anything that improved the world, but he wrote interesting about the possibilities and encouraged many others to per-sue technical careers. SF has often bad usage of logic, and the well known hero bias, or scientists that put together something to solve a current crisis that all their colleagues before had not managed to do. Unrealistic, but fun to read. SF writer write for a living, hard SF writers take stuff a bit more serial, but still are not actual experts on technology. Except when they are. Vinge would be such a case. Egan I have not read yet. Kurzweil seems to be one of the more present futurists, (Critiquing his ideas can take its own place.) But you will notice that the air gets pretty thin in this area, where everyone leads his own cult and spends more time on PR, than on finding good counter arguments for their current views. It would be awesome to have more people work on transhumanism/lifeextension/AI and what not, but that is not yet the case. There might even be good reasons for that, which LWers fail to perceive, or it could be that many scientists actually have a massive blindspot in regards to some of the topics. Regarding AI I fail to estimate how likely it is to reach it any time soon, since I really can not estimate all the complications on the way. The general possibility of human level intelligence looks plausible, because there are humans running around who have it. But even if the main goal of SIAI is never ever reached I already profit from the side products. Instead of concentrating on the AI stuff you can take the real-world part of the sequences and work on becoming a better thinker in whichever domain happens to be yours.

I don't think this post was well-written, at the least. I didn't even understand the tl;dr?

tldr; Is the SIAI evidence-based or merely following a certain philosophy? I'm currently unable to judge if the Less Wrong community and the SIAI are updating on fictional evidence or if the propositions, i.e. the basis for the strong arguments for action that are proclaimed on this site, are based on fact.

I don't see much precise expansion on this, except for MWI? There's a sequence on it.

And that is my problem. Given my current educational background and knowledge I cannot differentiate LW between a consistent internal logic, i.e. imagination or fiction and something which is sufficiently based on empirical criticism to provide a firm substantiation of the strong arguments for action that are proclaimed on this site.

Have you read the sequences?

As for why there aren't more people supporting SIAI, first of all, it's not widely known, second of all, it's liable to be dismissed on first impressions. Not many have examined the SIAI. Also, only (http://en.wikipedia.org/wiki/Religion#cite_ref-49)[4% of the general public in the US believe in neither a god nor a higher power]. The majority isn't always right.

I don't understand why this post has upvotes. It was unclear and seems topics went unresearched. The usefulness of donating to the SIAI has been discussed before, I think someone probably would've posted a link if asked in the open thread.

I was not sure whether to downvote this post for its epistemic value or upvote for instrumental (stimulating good discussion).

I ended up downvoting, I think this forum deserves better epistemic quality (I paused top-posting myself for this reason). I also donated to SIAI, because its value was once again validated to me by the discussion (though I have some reservations about apparent eccentricity of the SIAI folks, which is understandable (dropping out of high school is to me evidence of high rationality) but couterproductive (not having enough accepted academics involved). I mention this because it came up in the discussion and is definitely part of the subtext.

At to the concrete points of the post, I covered the part of it about the FAI vs AGI timeline here

The other part

Why is it that people like Vernor Vinge, Charles Stross or Ray Kurzweil are not running amok using all their influence to convince people of the risks ahead, or at least give all they have to the SIAI?

Is simply uninformed, and shows lack of diligence, which is the main reason I feel the post is not up to par and hope the clearly intelligent OP does some more homework and keeps contributing to the site.

• Vinge has written about bad Singularity scenarios (his Singularity paper and sci-fi).
• Stross has written about bad Singularity scenarios, at least in Accelerando (spoiler: humanity survives but only because AIs did not care about their resources at that point in time)
• Kurzweil has written about the possibility of bad scenarios (CIO article in discussion below)

I'll add one more, and to me rather damning: Peter Norvig, who wrote the (most widely used) book on AI and is head of research at Google is on the front page of SIAI (video clip), saying that as scientist we cannot ignore negative possibilities of AGI.

Do you have any reason to suppose that Charlie Stross has even considered SIAI's claims?

This post makes very weird claims regarding what SIAI's positions would be.

"Spend most on a particular future"? "Eliezer Yudkowsky is the right and only person who should be leading"?

It doesn't at all seem to me that stuff such as these would be SIAI's position. Why doesn't the poster provide references for these weird claims?

Here's a good reference for what SIAI's position actually is:

http://singinst.org/riskintro/index.html

Two key propositions seem to be:

1. The world is at risk from a superintelligence-gone-wrong;

2. The SIAI can help to do something about that.

Both propositions seem debatable. For the first point, certainly some scenarios are better than others - but the superintelligence causing widespread havoc by turning on its creators hypothesises substantial levels of incompetence, followed up by a complete failure of the surrounding advanced man-machine infrastructure to deal with the problem. Most humans may well have more to fear from a superintelligence-gone-right, but in dubious hands.

This was a very good job of taking a number of your comments and turning them into a coherent post. It raised my estimation that Eliezer will be able to do something similar with turning his blog posts into a book.

Greg Egan and the SIAI?

I completey forgot about this interview, so I already knew why Greg Egan isn't that worried:

… I think there’s a limit to this process of Copernican dethronement: I believe that humans have already crossed a threshold that, in a certain sense, puts us on an equal footing with any other being who has mastered abstract reasoning. There’s a notion in computing science of “Turing completeness”, which says that once a computer can perform a set of quite basic operations, it can be programmed to do absolutely any calculation that any other computer can do. Other computers might be faster, or have more memory, or have multiple processors running at the same time, but my 1988 Amiga 500 really could be programmed to do anything my 2008 iMac can do — apart from responding to external events in real time — if only I had the patience to sit and swap floppy disks all day long. I suspect that something broadly similar applies to minds and the class of things they can understand: other beings might think faster than us, or have easy access to a greater store of facts, but underlying both mental processes will be the same basic set of general-purpose tools. So if we ever did encounter those billion-year-old aliens, I’m sure they’d have plenty to tell us that we didn’t yet know — but given enough patience, and a very large notebook, I believe we’d still be able to come to grips with whatever they had to say.

Here's the Future of Humanity Institute's survey results from their Global Catastrophic Risks conference. The median estimate of extinction risk by 2100 is 19%, with 5% for AI-driven extinction by 2100:

http://www.fhi.ox.ac.uk/selected_outputs/fohi_publications/global_catastrophic_risks_survey

Unfortunately, the survey didn't ask for probabilities of AI development by 2100, so one can't get probability of catastrophe conditional on AI development from there.

Good thing at least some people here are willing to think critically.

I know these are unpopular views around here, but for the record:

• Risks be risks, but I believe it's unlikely that humanity will actually be destroyed in a foreseeable perspective.
• I do not think it's likely that we'll arrive at a superhuman AI during my lifetime, friendly or not.
• I do not think that Eliezer's techno-utopia is more desirable than simply humanity continuing to develop on its own at a natural pace.
• I do not fear death of old age, nor do I desire immortality or uploads.
• As muh as I respect Eliezer as a popularizer of science, when it comes to social wishes, he makes sweeping generalizations, too easily projects his personal desires onto the rest of humanity, and singles out whole broad categories as stupid or deluded just because they don't share his beliefs. If I don't trust his agenda enough to vote for him in a hypothetical election for President of United Earth, why should I trust his hypothetical AI?

Did you actually read through the MWI sequence before deciding that you still can't tell whether MWI is true because of (as I understand your post correctly) the state of the social evidence? If so, do you know what pluralistic ignorance is, and Asch's conformity experiment?

If you know all these things and you still can't tell that MWI is obviously true - a proposition far simpler than the argument for supporting SIAI - then we have here a question that is actually quite different from the one you seem to try to be presenting:

• I do not have sufficient g-factor to follow the detailed arguments on Less Wrong. What epistemic state is it rational for me to be in with respect to SIAI?

If you haven't read through the MWI sequence, read it. Then try to talk with your smart friends about it. You will soon learn that your smart friends and favorite SF writers are not remotely close to the rationality standards of Less Wrong, and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don't.