Can I say, first of all, that if you want to think realistically about a matter like this, you will have to find better authorities than science-fiction writers. Their ideas are generally not their own, but come from scientific and technological culture or from "futurologists" (who are also a very mixed bunch in terms of intellect, realism, and credibility); their stories present speculation or even falsehood as fact. It may be worthwhile going "cold turkey" on all the SF you have ever read, bearing in mind that it's all fiction that was ground out, word by word, by some human being living a very ordinary life, in a place and time not very far from you. Purge all the imaginary experience of transcendence from your system and see what's left.
Of course science-fictional thinking, treating favorite authors as gurus, and so forth is endemic in this subculture. The very name, "Singularity Institute", springs from science fiction. And SF occasionally gets things right. But it is far more a phenomenon of the time, a symptom of real things, rather than a key to understanding reality. Plain old science is a lot closer to being a reliable guide to reality, though even there - treating science as your authority - there are endless ways to go wrong.
A lot of the discourse here and in similar places is science fiction minus plot, characters, and other story-telling apparatus. Just the ideas - often the utopia of the hard-SF fan, bored by the human interactions and wanting to get on with the transcendent stuff. With transhumanist and singularity culture, this utopia has arrived, because you can talk all day about these radical futurist ideas without being tied to a particular author or oeuvre. The ideas have leapt from the page and invaded our brains, where they live even during the dull hours of daylight life. Hallelujah!
So, before you evaluate SIAI and its significance, there are a few more ideas that I would like you to drive from your brain: The many-worlds metaphysics. The idea of trillion-year lifespans. The idea that the future of the whole observable universe depends on the outcome of Earth's experiment with artificial intelligence. These are a few of the science-fiction or science-speculation ideas which have become a fixture in the local discourse.
I'm giving you this lecture because so many of your doubts about LW's favorite crypto-SF ideas masquerading as reality, are expressed in terms of ... what your favorite SF writers and futurist gurus think! But those people all have the same problem: they are trying to navigate issues where there simply aren't authorities yet. Stross and Egan have exactly the same syndrome affecting everyone here who writes about mind copies, superintelligence, alien utility functions, and so on. They live in two worlds, the boring everyday world and the world of their imagination. The fact that they produce descriptions of whole fictional worlds in order to communicate their ideas, rather than little Internet essays, and the fact that they earn a living doing this... I'm not sure if that means they have the syndrome more under control, or less under control, compared to the average LW contributor.
Probably you already know this, probably everyone here knows it. But it needs to be said, however clumsily: there is an enormous amount of guessing going on here, and it's not always recognized as such, and furthermore, there isn't much help we can get from established authorities, because we really are on new terrain. This is a time of firsts for the human species, both conceptually and materially.
Now I think I can start to get to the point. Suppose we entertain the idea of a future where none of these scenarios involving very big numbers (lifespan, future individuals, galaxies colonized, amount of good or evil accomplished) apply, and where none of these exciting info-metaphysical ontologies turns out to be correct. A future which mostly remains limited in the way that all human history to date has been limited, limited in the ways which inspire such angst and such promethean determination to change things, or determination to survive until they change, among people who have caught the singularity fever. A future where everyone is still going to die, where the human race and its successors only last a few thousand years, not millions or billions of them. If that is the future, could SIAI still matter?
My answer is yes, because artificial intelligence still matters in such a future. For the sake of argument, I may have just poured cold water on a lot of popular ideas of transcendence, but to go further and say that only natural life and natural intelligence will ever exist really would be obtuse. If we do accept that "human-level" artificial intelligence is possible and is going to happen, then it is a matter at least as consequential as the possibility of genocide or total war. Ignoring, again for the sake of a limited argument, all the ideas about planet-sized AIs and superintelligence, and it's still easy to see that AI which can out-think human beings and which has no interest in their survival ought to be possible. So even in this humbler futurology, AI is still an extinction risk.
The solution to the problem of unfriendly AI most associated with SIAI - producing the coherent extrapolated volition of the human race - is really a solution tailored to the idea of a single super-AI which undergoes a "hard takeoff", a rapid advancement in power. But SIAI is about a lot more than researching, promoting, and implementing CEV. There's really no organization like it in the whole sphere of "robo-ethics" and "ethical AI". The connection that has been made between "friendliness" and the (still scientifically unknown) complexities of the human decision-making process is a golden insight that has already justified SIAI's existence and funding many times over. And of course SIAI organizes the summits, and fosters a culture of discussion, both in real life and online (right here), which is a lot broader than SIAI's particular prescriptions.
So despite the excesses and enthusiasms of SIAI's advocates, supporters, and leading personalities, it really is the best thing we have going when it comes to the problem of unfriendly AI. Whether and how you personally should be involved with its work - only you can make that decision. (Even constructive criticism is a way of helping.) But SIAI is definitely needed.
I hadn't thought of it this way, but on reflection of course it's true.
Major update here.
The state of affairs regarding the SIAI and its underlying rationale and rules of operation are insufficiently clear.
Most of the arguments involve a few propositions and the use of probability and utility calculations to legitimate action. Here much is uncertain to an extent that I'm not able to judge any nested probability estimations. Even if you tell me, where is the data on which you base those estimations?
There seems to be an highly complicated framework of estimations to support and reinforce each other. I'm not sure how you call this in English, but in German I'd call that a castle in the air.
I know that what I'm saying may simply be due to a lack of knowledge and education, that is why I am inquiring about it. How many of you, who currently support the SIAI, are able to analyse the reasoning that led you to support the SIAI in the first place, or at least substantiate your estimations with other kinds of evidence than a coherent internal logic?
I can follow much of the reasoning and arguments on this site. But I'm currently unable to judge their overall credence. Are the conclusions justified? Is the coherent framework build around the SIAI based on firm ground? There seems to be no critical inspection or examination by a third party. There is no peer review. Yet people are willing to donate considerable amounts of money.
I'm concerned that, although consistently so, the SIAI and its supporters are updating on fictional evidence. This post is meant to inquire about the foundations of your basic premises. Are you creating models to treat subsequent models or are your propositions based on fact?
An example here is the use of the Many-worlds interpretation. Itself a logical implication, can it be used to make further inferences and estimations without additional evidence? MWI might be the only consistent non-magic interpretation of quantum mechanics. The problem here is that such conclusions are, I believe, widely considered not to be enough to base further speculations and estimations on. Isn't that similar to what you are doing when speculating about the possibility of superhuman AI and its consequences? What I'm trying to say here is that if the cornerstone of your argumentation, if one of your basic tenets is the likelihood of superhuman AI, although a valid speculation given what we know about reality, you are already in over your head with debt. Debt in the form of other kinds of evidence. Not to say that it is a false hypothesis, that it is not even wrong, but that you cannot base a whole movement and a huge framework of further inference and supportive argumentation on such premises, on ideas that are themselves not based on firm ground.
The gist of the matter is that a coherent and consistent framework of sound argumentation based on unsupported inference is nothing more than its description implies. It is fiction. Imagination allows for endless possibilities while scientific evidence provides hints of what might be possible and what impossible. Science does provide the ability to assess your data. Any hint that empirical criticism provides gives you new information on which you can build on. Not because it bears truth value but because it gives you an idea of what might be possible. An opportunity to try something. There’s that which seemingly fails or contradicts itself and that which seems to work and is consistent.
And that is my problem. Given my current educational background and knowledge I cannot differentiate LW between a consistent internal logic, i.e. imagination or fiction, and something which is sufficiently based on empirical criticism to provide a firm substantiation of the strong arguments for action that are proclaimed by the SIAI.
Further, do you have an explanation for the circumstance that Eliezer Yudkowsky is the only semi-popular person who's aware of something that might shatter the universe? Why is it that people like Vernor Vinge, Robin Hanson or Ray Kurzweil are not running amok using all their influence to convince people of the risks ahead, or at least give all they have to the SIAI? Why aren't Eric Drexler, Gary Drescher or AI researches like Marvin Minsky worried to the extent that they signal their support?
I'm talking to quite a few educated people outside this community. They do not doubt all those claims for no particular reason. Rather they tell me that there are too many open questions to focus on the possibilities depicted by the SIAI and to neglect other near-term risks that might wipe us out as well.
I believe that many people out there know a lot more than I do, so far, about related topics and yet they seem not to be nearly as concerned about the relevant issues than the average Less Wrong member. I could have named other people. That's besides the point though, it's not just Hanson or Vinge but everyone versus Eliezer Yudkowsky and some unknown followers. What about the other Bayesians out there? Are they simply not as literate as Eliezer Yudkowsky in the maths or maybe somehow teach but not use their own methods of reasoning and decision making?
What do you expect me to do, just believe Eliezer Yudkowsky? Like I believed so much in the past which made sense but turned out to be wrong? Maybe after a few years of study I'll know more.
...
2011-01-06: As this post received over 500 comments I am reluctant to delete it. But I feel that it is outdated and that I could do much better today. This post has however been slightly improved to account for some shortcomings but has not been completely rewritten, neither have its conclusions been changed. Please account for this when reading comments that were written before this update.
2012-08-04: A list of some of my critical posts can be found here: SIAI/lesswrong Critiques: Index