we should assume there are already a large number of unfriendly AIs in the universe, and probably in our galaxy; and that they will assimilate us within a few million years.

Let's be Bayesian about this.

Observation: Earth has not been assimilated by UFAIs at any point in the last billion years or so. Otherwise life on Earth would be detectably different.

It is unlikely that there are no/few UFAIs in our galaxy/universe, but if they do exist it is unlikely that they would not already have assimilated us.

I don't have enough information to give exact probabilities, but it's a lot more likely than you seem to think that we will survive the next billion years without assimilation from an alien UFAI.

Personally, I think the most likely scenario is either that Earth is somehow special and intelligent life is rarer than we give it credit for; or that alien UFAIs are generally not interested in interstellar/intergalactic travel.

EDIT: More rigorously, let Uf be the event "Alien UFAIs are a threat to us", and Ap be the event "We exist today" (anthropic principle). The prior probability P(Uf) is large, by your arguments, but P(Ap given Uf) is much smaller than P(Ap given not-Uf). Since we observe Ap to be true, the posterior probability P(Uf given Ap) is fairly small.

If we work from assumptions that make it likely for the universe to contain a "large number" of natural intelligences that go on to build UFAIs that assimilate on an interstellar or intergalactic level, then Earth would almost certainly have already been assimilated millions, even billions of years ago, and we accordingly would not be around to theorize.

After all, one needs only one species to make a single intergalacitc assimilating UFAI emerge somewhere in the Virgo Supercluster more than 110 million years ago to have assimilated the whole supercluster by now, using no physics we don't already know.

Unless what we believe about evolution is wrong, or we are very very lucky, we should assume there are already a large number of unfriendly AIs in the universe, and probably in our galaxy.

Doesn't this depend on the likelihood/prevalence of intelligent life? For all we know, we might be the only sentient species out there. Also, even if there are a lot o unfriendly AIs out there, a friendly one would still vastly improve our fate, whether by fighting off the unfriendly ones, or reaching an agreement with them to mutual benefit, or rescue simulations. The greater age of the alien ones might give them a massive advantage, but that depends on whether FOOM will ever run into diminishing returns. If it does, then the difference between, say, a 500,000 year old AI and a 2 million year old AI may not be much.

None of this should be construed as supporting the arguments in "Put all your eggs in one basket".

OK, this is my first day posting on Less Wrong. This topic "Don't Plan For the Future" interests me a lot and I have a few ideas on it. Yet it's been inactive for over a year. Possibilities that occur to me: (1) the subject has been taken up more definitively in a more recent thread, and I need to find it, (2) because of the time lag, I should start a new "Discussion" (I think I have more than 2 karma points already so it's at least possible) even if it's the same basic idea as this, (3) I should post ideas and considerations right here despite the time lag. If there's some guide that would answer this question, I'll happily take a pointer to that as well.

As I understand the usage elsewhere on this site, a Friendly AI created by nonhumans ought to embody the terminal values of the creating race, just as we talk about FAIs created by humans embodying the terminal values of humans.

And presumably the same reasoning that concludes that a self-optimizing AI, unless created taking exquisite care to ensure Friendliness, won't actually be compatible with human values (due to the Vastness of mindspace and so forth), also concludes that a powerful alien race, having necessarily been created without taking such care, will similarly not be compatible with human values.

This line of reasoning seems to conclude that drawing a distinction between alien UFAIs and alien FAIs (and, for that matter, alien NIs) is moot in this context -- they are all a threat to humanity.

Which, yes, leads to exactly the same default assumption you cite.

The universe may also contain a large number of rapidly expanding friendly AIs. The likelihood of one arising on this planet may correlate with the likelihood of it arising on other planets, although I'm not sure how strong a correlation we can suppose for intelligent life forms with completely different evolutionary histories. In that case, anything that increases the chance of friendly AI arising on this planet can also be taken to decrease our chance of being subsumed by an extraplanetary unfriendly AI.

An AI that is friendly to one intelligent species might not be friendly to others, but such AI should probably be considered to be imperfectly friendly.

After I learned of MWI I felt the Fermi Paradox was more or less solved. Considering the number of near-misses we've already had in global nuclear war it seems likely that almost all intelligent species destroy themselves shortly after learning to crack the atom (including us). The universes where a single species managed to avoid self-annihilation will greatly out number the universes where multiple species managed to do so.

Assuming quantum fluctuations can have macro-level effects, of course.

Should we devote resources to trying to expand across the galaxy and thus influence events millions of years in the future? I say no.

I've been thinking about this question for many years, and it's just in the past few days I've learned about the Singularity. I don't at the moment assign a very high probability to that -- yes, I'm ignorant, but I'm going to leave it out for the moment.

Suppose we posit that from some currently unavailable combination of technology, physics, psychology, politics and economics (for starters) we can have "legs" and cover interstellar ground. We also crucially need a density of planets that can be exploited to create the vibrant economies that could launch other expensive spacecraft to fuel exponential growth. If we're going to expand using humans, we have to assume a rather high density of not just planets that can support intelligent life, but planets that can support our particular form of intelligent life -- earth-like planets. We have to assume that those planets have not evolved competent, intelligent life of their own -- even if they are far behind us technologically, their inherent advantages of logistics could very well keep us from displacing them. But on the plus side, it also seems highly likely that if we can get such a process of exponential growth going in our corner of the galaxy, it could then be expanded throughout our galaxy (at the least).

If we can do it, so can they -- actually, they already did.

To expand that, I attach great importance to the fallacy of human exceptionalism. Over history we've had to give up beliefs about cultural and racial superiority, humans being fundamentally different from animals, the earth being the center of the universe, the sun being the center of the universe... The list is familiar.

We've discovered stars with planets. Perhaps fewer have small, rocky (non-gas giant) planets than theories initially suggested, but there are a few (last I knew) and that's just a small adjustment in our calculations. We have no evidence whatsoever that our solar system is exceptional on the scale of the galaxy -- there are surely many millions of rocky planets (a recent news story suggests billions).

Just how improbable is the development of intelligent life? I'd be interested to know how much deep expertise in biology we have in this group. The 2011 survey results say 174 people (16%) in the hard sciences, with some small fraction of that biologists? I claim no expertise, but can only offer what (I think) I know.

First, I'd heard it guessed that life developed on earth just about as soon as the earth was cool enough to allow its survival. Second, evolution has produced many of its features multiple times. This seems to bear on how likely evolution elsewhere is to develop various characteristics. If complicated ones like wings and eyes and (a fair amount of) intelligence evolved independently several times, then it wasn't just some miraculous fluke. It makes such developments in life on other planets seem far more probable. Third, the current time in earth history does not have a special status. If intelligent life hadn't evolved on earth now, it had a few billion more years to happen.

Based on those considerations, I consider it a near certainty that alien civilizations have developed -- I'd guess many thousands in our galaxy as a minimum. It's a familiar argument that we should assume we are in the middle of such a pack temporally, so at the least hundreds of civilizations started millions of years ago. If expansion across the galaxy was possible, they'd be here by now. The fact that we have detected no signals from SETI says virtually nothing -- that just means there is nobody in our immediate vicinity who is broadcasting right now.

Since we haven't observed any alien presence on earth, we would have to assume that civilization expansion is not independent -- some dominant civilization suppresses others. There are various possibilities as to the characteristics of that one civilization. They might want to remain hidden. They might not interfere until a civilization grows powerful enough to start sending out colonies to other worlds. Perhaps they just observe us indefinitely and only interfere if we threaten their own values. Even in some benign confederation, where all the civilizations share what they have to offer, we would offer just one tiny drop to a bucket formed from -- what, millions? -- of other civilizations. What all of these have in common is that it is not our values that dominate the future: it's theirs.

It seems likely to me that my initial assumption about exponential space colonization is wrong. It is unfashionable in futurist circles to suggest something is impossible, especially something like sending colonists to other planets, something that doesn't actually require updates to our understanding of the laws of physics. Critics point out all the other times someone said something was impossible, and it turned out that it could be done. But that is very different from saying that everything that seems remotely plausible can in fact be done. If I argued against interstellar colonization based on technical difficulties, that would be a weak argument. My argument is based on the fact that if it were possible, the other civilizations would be here already.

This argument extends to the colonization potential of robots produced in the aftermath of the Singularity. If their robots could do it, they'd be here already.

To achieve the huge win that would make such an expensive, difficult project worthwhile, exponential space colonization has to be possible, and we have to be the first ones. I think both are separately highly unlikely, and in combination astronomically unlikely.