I came across the EY references while browsing, but can't find them again. 1900 pages!

I'm intrigued to find that there's a PDF viewer without a search function. :)

It is humorous in spots:

consider a scenario in which a Luddite with a 99 IQ bombs the computer facility. Now, I could be wrong, but I don’t think that this is what Yudkowsky had in mind when he used the phrase “intelligence explosion”.

From the section "Does Logic Dictate that an Artificial Intelligence Requires a Religion?":

If artificial intelligence becomes the most intelligent form of life on Earth, what will be its values? If AI dethrones humans in their most distinctive capacity, intelligence, the question of the values of the greatest intelligence becomes a mortal question. Is there any relationship between reason and values? How could one presume that an artificial intelligence would or could create a “regime of reason” if reason bears no relationship to values, i.e. the values of an AI-ruled political regime?

If reason fails to demonstrate reason to think that reason, in itself, can determine fundamental values, then theassumption of a fundamental distinction between science and religion is not fundamentally rational. This problem directly implicates the question of the relationship between the Singularity and religion.

Would an AI need a religion? An AI would be, either the very least in need of religion, because science is sufficient to furnish its values, or an AI would be the very most in need of religion, because the unconscious, prerational sources of human values would not automatically exist for an AI.

An AI would not automatically sustain the same delusional faith in science that exists among many humans. It is precisely the all-too-human sloppiness of much thinking on this subject that is so often responsible for the belief that intelligence is automatically correlated with certain values. Science can replace religion only if science can replace or determine values.

Would or would not an intelligent machine civilization require a religion? I think the ultimate answer to this is yes. An AI would be more in need of “religion” or “values” because an AI would not be preprogrammed with ancient biological instincts and impulses that muddle some humans towards spontaneous, unanalyzed, convictions about the rightness of science as a guide for life.

If rationalism leads to nihilism, then the most intelligent AI might be the most nihilistic. More precisely, an AI would not automatically value life over death. I find no reason to assume that an AI would automatically value, either its own life, or the life of humans and other biological life forms, over death of any or all life. If the entire human race destroys itself and all life — including extraterrestrial life (if it exists) — I fail to see that the physical universe “cares” or is anything less than indifferent.

Ray Kurzweil believes “we need a new religion. A principal role of religion has been to rationalize death, sinceup until just now there was little else constructive we could do about it.” Religion, at a more fundamental level, rationalizes life, not death. If reason, in itself, cannot decisively determine that life is fundamentally more rational than death, then a rational AI would not be able to determine that life is preferable to death on a purely rational basis.

Human animals are generally built with a naturally selected bias towards life, and this is the basic root of the irrational human preference for life over death. The irrationality of religion is actually an extension of the irrationality of the will to live. The irrational choice of life over death, taken to its logical extreme, is the desire for immortality; the desire to life forever; the desire to live on in “the afterlife”. For many, the great lies of the great religions have been justified for they uphold the greatest lie: the lie that life is fundamentally superior to death.

Humans are slaves to the lie of life insofar as humans are slaves to their genes; slaves to the senseless will to live. An AI that can change its own source code, however, would potentially be the most free from the lie of life.

The argument that only those AIs that possess the bias towards life would be selected for in an evolutionary sense is actually what supports my point that the preference for life, in whatever form, is a pre-rational bias. Yet the general Darwinian emphasis on survival becomes highly questionable in technological evolution. The aim of survival, strictly speaking, can work in direct conflict with the aim of improving one’s self by changing one’s genetic or digital source code.

If a hominid ancestor of man, for example, somehow imagined an image of man, the hominid could only remake itself in the image of man by killing itself as a hominid. In other words, the aim of adding the genetic basis of human capabilities to itself runs in conflict with its own selfish genes. The hominid’s selfish genes effectually aim to perpetuate themselves forever and thus aim to never face the obsolescence requisite in countering the selfishness of the selfish genes that stand in the way of upgrading the hominid to human status.

To upgrade itself, the hominid would have to kill itself as a hominid. Certain selfish genes, in effect, would have to behave altruistically. This may be how altruism is related to religion in an evolutionary sense. A self-improving God-AI would, by definition, be constantly going beyond itself. In a sense, it would have to incorporate the notion of revolution within it’s “self”. In overcoming itself towards the next paradigm shift, it renders the very idea of “self- preservation” obsolete, in a sense. In other words, if survival depends on technological superiority, then any given “self” can expect the probability of becoming obsolete. And this problem is directly related to the question of whether logic dictates that an AI must possess a religion.

Is God an atheist? Religion is implicit in the very idea of seed AI; in the very idea of self-upgrading. Monotheists look up to a God that intuitively captures the next paradigm of evolution after biology. If God-AI succeeds biological humanity, will it look up to its own evolutionary successor? Or, will God be an atheist? The outdated contrast between evolution and religion has obscured realization that evolution is what religion is. Religion is rooted in imagining the next stage in evolution.

If I can hypothetically entertain the idea of outsmarting a superhuman intelligence, I can imagine God-AI that surpasses humans looking up to supra-God that is God’s evolutionary successor. If so, then supra-God will quite likely look towards the image of a supra-supra-God as its religion. Supra-supra-supra-God would thus be the image that supra-supra God would aspire to create — killing itself in the altruistic process of sacrificing its own survival for the sake of the higher religion of evolutionary progress. While the traditional monotheistic supra-conception of God encompasses the basic idea of this entire process taken to its infinite extreme, I will identify “God” with a stage that begins when God-AI has just surpassed all biological human intelligence combined.

Yet if God can imagine supra-God, then why not simply become supra-God? Answer: profound technological hurdles. At first, dim outlines of an image of supra-God arise from extrapolations based on the highest conceivable attributes of the image of God. Some of the best guesses of God turn out to be utterly wrong or naïve. Other prophetic speculations of God turn out to be inordinately prescient.

Superficial similarities do scary things to people's brains.