After complaints about misquoting, a slightly altered version of my AI 5-minute talk is now up at:

http://www.youtube.com/watch?v=3jSMe0owGMs

This is a summary of material from various posts and discussions. My thanks to Eliezer Yudkowsky, Daniel Dewey, Paul Christiano, Nick Beckstead, and several others.

Several ideas have been floating around LessWrong that can be organized under one concept, relating to a subset of AI safety problems. I’d like to gather these ideas in one place so they can be discussed as a unified concept. To give a definition:

A computation hazard is a large negative consequence that may arise merely from vast amounts of computation, such as in a future supercomputer.

For example, suppose a computer program needs to model people very accurately to make some predictions, and it models those people so accurately that the "simulated" people can experience conscious suffering. In a very large computation of this type, millions of people could be created, suffer for some time, and then be destroyed when they are no longer needed for making the predictions desired by the program. This idea was first mentioned by Eliezer Yudkowsky in Nonperson Predicates.

There are other hazards that may arise in the course of running large-scale computations. In general, we might say that:

Large amounts of computation will likely consist in running many diverse algorithms. Many algorithms are computation hazards. Therefore, all else equal, the larger the computation, the more likely it is to produce a computation hazard.

Of course, most algorithms may be morally neutral. Furthermore, algorithms must be somewhat complex before they could possibly be a hazard. For instance, it is intuitively clear that no eight-bit program could possibly be a computation hazard on a normal computer. Worrying computations therefore fall into two categories: computations that run most algorithms, and computations that are particularly likely to run algorithms that are computation hazards.

An example of a computation that runs most algorithms is a mathematical formalism called Solomonoff induction. First published in 1964, it is an attempt to formalize the scientific process of induction using the theory of Turing machines. It is a brute-force method that finds hypotheses to explain data by testing all possible hypotheses. Many of these hypotheses may be algorithms that describe the functioning of people. At a sufficient precision, these algorithms themselves may experience consciousness and suffering. Taken literally, Solomonoff induction runs all algorithms; therefore it produces all possible computation hazards. If we are to avoid computation hazards, any implemented approximations of Solomonoff induction will need to determine ahead of time which algorithms are computation hazards.

Computations that run most algorithms could also hide in other places. Imagine a supercomputer’s power is being tested on a simple game, like chess or Go. The testing program simply tries all possible strategies, according to some enumeration. The best strategy that the supercomputer finds would be a measure of how many computations it could perform, compared to other computers that ran the same program. If the rules of the game are complex enough to be Turing complete (a surprisingly easy achievement) then this game-playing program would eventually simulate all algorithms, including ones with moral status.

Of course, running most algorithms is quite infeasible simply because of the vast number of possible algorithms. Depending on the fraction of algorithms that are computation hazards, it may be enough that a computation run an enormous number which act as a random sample of all algorithms. Computations of this type might include evolutionary programs, which are blind to the types of algorithms they run until the results are evaluated for fitness. Or they may be Monte Carlo approximations of massive computations.

But if computation hazards are relatively rare, then it will still be unlikely for large-scale computations to stumble across them unguided. Several computations may fall into the second category of computations that are particularly likely to run algorithms that are computation hazards. Here we focus on three types of computations in particular: agents, predictors and oracles. The last two types are especially important because they are often considered safer types of AI than agent-based AI architectures. First I will stipulate definitions for these three types of computations, and then I will discuss the types of computation hazards they may produce.

Agents

An agent is a computation which decides between possible actions based on the consequences of those actions. They can be thought of as “steering” the future towards some target, or as selecting a future from the set of possible futures. Therefore they can also be thought of as having a goal, or as maximizing a utility function.

Sufficiently powerful agents are extremely powerful because they constitute a feedback loop. Well-known from physics, feedback loops often change their surroundings incredibly quickly and dramatically. Examples include the growth of biological populations, and nuclear reactions. Feedback loops are dangerous if their target is undesirable. Agents will be feedback loops as soon as they are able to improve their ability to improve their ability to move towards their goal. For example, humans can improve their ability to move towards their goal by using their intelligence to make decisions. A student aiming to create cures can use her intelligence to learn chemistry, therefore improving her ability to decide what to study next. But presently, humans cannot improve their intelligence, which would improve their ability to improve their ability to make decisions. The student cannot yet learn how to modify her brain in order for her to more quickly learn subjects.

Predictors

A predictor is a computation which takes data as input, and predicts what data will come next. An example would be certain types of trained neural networks, or any approximation of Solomonoff induction. Intuitively, this feels safer than an agent AI because predictors do not seem to have goals or take actions; they just report predictions as requested by human.

Oracles

An oracle is a computation which takes questions as input, and returns answers. They are broader than predictors in that one could ask an oracle about predictions. Similar to a predictor, oracles do not seem to have goals or take actions. (Some material summarized here.)

Examples of hazards

Agent-like computations are the most clearly dangerous computation hazards. If any large computation starts running the beginning of a self-improving agent computation, it is difficult to say how far the agent may safely be run before it is a computation hazard. As soon as the agent is sufficiently intelligent, it will attempt to acquire more resources like computing substrate and energy. It may also attempt to free itself from control of the parent computation.

Another major concern is that, because people are an important part of the surroundings, even non-agent predictors or oracles will simulate people in order to make predictions or give answers respectively. Someone could ask a predictor, “What will this engineer do if we give him a contract?” It may be that the easiest way for the predictor to determine the answer is to simulate the internal workings of the given engineer's mind. If these simulations are sufficiently precise, then they will be people in and of themselves. The simulations could cause those people to suffer, and will likely kill them by ending the simulation when the prediction or answer is given.

Similarly, one can imagine that a predictor or oracle might simulate powerful agents; that is, algorithms which efficiently maximize some utility function. Agents may be simulated because many agent-like entities exist in the real world, and their behavior would need to be modeled. Or, perhaps oracles would investigate agents for the purpose of answering questions better. These agents, while being simulated, may have goals that require acting independently of the oracle. These agents may also be more powerful than the oracles, especially since the oracles were not designed with self-improvement behavior in mind. Therefore these agents may attempt to “unbox” themselves from the simulation and begin controlling the rest of the universe. For instance, the agents may use previous questions given to the oracle to deduce the nature of the universe and the psychology of the oracle-creators. (For a fictional example, see That Alien Message.) Or, the agent might somehow distort the output of the predictor, in a way that what the oracle predicts will cause us to unbox the agent.

Predictors also have the problem of self-fulfilling prophecies (first suggested here). An arbitrarily accurate predictor will know that its prediction will affect the future. Therefore, to be a correct prediction, it must make sure that delivering its prediction doesn’t cause the receiver to act in a way that negates the prediction. Therefore, the predictor may have to choose between predictions which cause the receiver to act in a way that fulfills the prediction. This is a type of control over the user. Since the predictor is super-intelligent, any control may rapidly optimize the universe towards some unknown goal.

Overall, there is a large worry that sufficiently intelligent oracles or predictors may become agents. Beside the above possibilities, some are worried that intelligence is inherently an optimization process, and therefore oracles and predictors are inherently satisfying some utility function. This, combined with the fact that nothing can be causally isolated from the rest of the universe, seems to invite an eventual AI-takeoff.

Methods for avoiding computational hazards

It is often thought that, while no proposal has yet been shown safe from computational hazards, oracles and predictors are safer than deliberately agent-based AGI. Other methods have been proposed to make these even safer. Armstrong et al. describe many AI safety measures in general. Below we review some possible techniques for avoiding computational hazards specifically.

One obvious safety practice is to limit the complexity, or the size of computations. In general, this will also limit the algorithm below general intelligence, but it is a good step while progressing towards FAI. Indeed, it is clear that all current prediction or AI systems are too simple to either be general intelligences, or pose as a computational hazard.

A proposal for regulating complex oracles or predictors is to develop safety indicators. That is, develop some function that will evaluate the proposed algorithm or model, and return whether it is potentially dangerous. For instance, one could write a simple program that rejects running an algorithm if any part of it is isomorphic to the human genome (since DNA clearly creates general intelligence and people under the right circumstances). Or, to measure the impact of an action suggested by an oracle, one could ask how many humans would be alive one year after the action was taken.

But one could only run an algorithm if they were sure it was not a person. A function that could evaluate an algorithm and return 0 only if it is not a person is called a nonperson predicate. Some algorithms are obviously not people. For example, squaring the numbers from 1 to 100 will not simulate people. Any algorithm whose behavior is periodic with a short period is unlikely to be a person, or nearly any presently constructed software. But in general this seems extremely difficult to verify. It could be that writing nonperson predicates or other safety indicators is FAI-complete in that sense that if we solve them, we will have discovered friendliness theory. Furthermore, it may be that some attempts to evaluate whether an algorithm is a person actually causes a simulation of a person, by running parts of the algorithm, by modeling a person for comparison, or by other means. Similarly, it may be that attempts to investigate the friendliness of a particular agent cause that agent to unbox itself.

Predictors seem to be one of the most goal-agnostic forms of AGI. This makes them a very attractive model in which to perfect safety. Some ideas for avoiding self-fulfilling predictions suggest that we ask the predictor to tell us what it would have predicted if we hadn’t asked (first suggested here). This frees the predictor from requiring itself to make predictions consistent with our behavior. Whether this will work depends on the exact process of the predictor; it may be so accurate that it cannot deal with counterfactuals, and will simply report that it would have predicted that we would have asked anyway. It is also problematic that the prediction is now inaccurate; because it has told us, we will act, possibly voiding any part of the prediction.

A very plausible but non-formal solution is to aim for a soft takeoff. For example, we could build a predictor that is not generally intelligent, and use it to investigate safe ways advance the situation. Perhaps we could use a sub-general intelligence to safely improve our own intelligence.

Have I missed any major examples in this post? Does “computation hazards” seem like a valid concept as distinct from other types of AI-risks?

References

Armstrong S., Sandberg A., Bostrom N. (2012). “Thinking inside the box: using and controlling an Oracle AI”. Minds and Machines, forthcoming.

Solomonoff, R., "A Formal Theory of Inductive Inference, Part I" Information and Control, Vol 7, No. 1 pp 1-22, March 1964.

Solomonoff, R., "A Formal Theory of Inductive Inference, Part II" Information and Control, Vol 7, No. 2 pp 224-254, June 1964.

As everyone here knows, it would be a stupid idea to switch from airplanes to cars out of safety/terrorism concerns: Cars are a much more risky means of transportation than airplanes. But what other major risks are there that many people systematically undervalue or are not even consciously aware of?

The same can be asked for chances.

It's just occurred to me that, giving all the cheerful risk stuff I work with, one of the most optimistic things people could say to me would be:

"You've wasted your life. Nothing of what you've done is relevant or useful."

That would make me very happy. Of course, that only works if it's credible.

All the posts on FAI theory as of late have given me cause to think. There's something in the conversations about it that has always bugged me, but it is something that I haven't found the words for before now.

It is something like this:

Say that you manage to construct an algorithm for FAI...

Say that you can show that it isn't going to be a dangerous mistake...

And say you do all of this, and popularize it, before AGI is created (or at least, before an AGI goes *FOOM*)...

...

How in the name of Sagan are you actually going to ENFORCE the idea that all AGIs are FAIs? 

I mean, if it required some rare material (like nuclear weapons) or large laboratories (like biological wmds) or some other resource that you could at least make artificially scarce, you could set up a body that ensures that any AGI created is an FAI.

But if all it is, is the right algorithms, the right code, and enough computing power... even if you design a theory for FAI, how would you keep someone from making UFAI anyway? Between people experimenting with the principles (once known), making mistakes, and the prospect of actively malicious *humans*... it just seems like unless you somehow come up with an internal mechanism that makes FAI better and stronger than any UFAI could be, and the solution turns out to be such that any idiot could see that it was a better solution... that UFAI is going to exist at some point no matter what.

At that point, it seems like the question becomes not "How do we make FAI?" (although that might be a secondary question) but rather "How do we prevent the creation of, eliminate, or reduce potential damage from UFAI?" Now, it seems like FAI might be one thing that you do toward that goal, but if UFAI is a highly likely consequence of AGI even *with* an FAI theory, shouldn't the focus be on how to contain a UFAI event?

A short article, quoting me and Luke:

http://www.wired.co.uk/news/archive/2012-05/17/the-dangers-of-an-ai-smarter-than-us

It makes the point that it's not the shambling robots that are the risks here, but the other powers of intelligence.

I did a talk at the 25th Oxford Geek night, in which I had five minutes to present the dangers of AI. The talk is now online. Though it doesn't contain anything people at Less Wrong would find new, I feel it does a reasonable job at pitching some of the arguments in a very brief format.

LessWrong is not big on discussion of non-AI existential risks. But Neil deGrasse Tyson notes killer asteroids not just as a generic problem, but as a specific one, naming Apophis as an imminent hazard.

So treat this as your exercise for today: what are the numbers, what is the risk, what are the costs, what actions are appropriate? Assume your answers need to work in the context of a society that's responded to the notion of anthropogenic climate change with almost nothing but blue vs. green politics.

The Power of Algorithms

Evolutionary processes are the most evident example of the power of simple algorithms [1][2][3][4][5].

The field of evolutionary biology gathered a vast amount of evidence [6] that established evolution as the process that explains the local decrease in entropy [7], the complexity of life.

Since it can be conclusively shown that all life is an effect of an evolutionary process it is implicit that everything we do not understand about living beings is also an effect of evolution.

We might not understand the nature of intelligence [8] and consciousness [9] but we do know that they are the result of an optimization process that is neither intelligent nor conscious.

Therefore we know that it is possible for an physical optimization process to culminate in the creation of more advanced processes that feature superior qualities.

One of these qualities is the human ability to observe and improve the optimization process that created us. The most obvious example being science [10].

Science can be thought of as civilization-level self-improvement method. It allows us to work together in a systematic and efficient way and accelerate the rate at which further improvements are made.

The Automation of Science

We know that optimization processes that can create improved versions of themselves are possible, even without an explicit understanding of their own workings, as exemplified by natural selection.

We know that optimization processes can lead to self-reinforcing improvements, as exemplified by the adaptation of the scientific method [11] as an improved evolutionary process and successor of natural selection.

Which raises questions about the continuation of this self-reinforcing feedback cycle and its possible implications.

One possibility is to automate science [12][13] and apply it to itself and its improvement.

But science is a tool and its bottleneck are its users. Humans, the biased [14] effect of the blind idiot god that is evolution.

Therefore the next logical step is to use science to figure out how to replace humans by a better version of themselves, artificial general intelligence.

Artificial general intelligence, that can recursively optimize itself [15], is the logical endpoint of various converging and self-reinforcing feedback cycles.

Risks from AI

Will we be able to build an artificial general intelligence? Yes, sooner or later.

Even the unintelligent, unconscious and aimless process of natural selection was capable of creating goal-oriented, intelligent and conscious agents that can think ahead, jump fitness gaps and improve upon the process that created them to engage in prediction and direct experimentation.

The question is, what are the possible implications of the invention of an artificial, fully autonomous, intelligent and goal-oriented optimization process?

One good bet is that such an agent will recursively improve its most versatile, and therefore instrumentally useful, resource. It will improve its general intelligence, respectively cross-domain optimization power.

Since it is unlikely that human intelligence is the optimum, the positive feedback effect, that is a result of using intelligence amplifications to amplify intelligence, is likely to lead to a level of intelligence that is generally more capable than the human intelligence level.

Humans are unlikely to be the most efficient thinkers because evolution is mindless and has no goals. Evolution did not actively try to create the smartest thing possible.

Evolution is further not limitlessly creative, each step of an evolutionary design must increase the fitness of its host. Which makes it probable that there are artificial mind designs that can do what no product of natural selection could accomplish, since an intelligent artificer does not rely on the incremental fitness of each step in the development process.

It is actually possible that human general intelligence is the bare minimum. Because the human level of intelligence might have been sufficient to both survive and reproduce and that therefore no further evolutionary pressure existed to select for even higher levels of general intelligence.

The implications of this possibility might be the creation of an intelligent agent that is more capable than humans in every sense. Maybe because it does directly employ superior approximations of our best formal methods, that tell us how to update based on evidence and how to choose between various actions. Or maybe it will simply think faster. It doesn’t matter.

What matters is that a superior intellect is probable and that it will be better than us at discovering knowledge and inventing new technology. Technology that will make it even more powerful and likely invincible.

And that is the problem. We might be unable to control such a superior being. Just like a group of chimpanzees is unable to stop a company from clearing its forest [16].

But even if such a being is only slightly more capable than us. We might find ourselves at its mercy nonetheless.

Human history provides us with many examples [17][18][19] that make it abundantly clear that even the slightest advance can enable one group to dominate others.

What happens is that the dominant group imposes its values on the others. Which in turn raises the question of what values an artificial general intelligence might have and the implications of those values for us.

Due to our evolutionary origins, our struggle for survival and the necessity to cooperate with other agents, we are equipped with many values and a concern for the welfare of others [20].

The information theoretic complexity [21][22] of our values is very high. Which means that it is highly unlikely for similar values to automatically arise in agents that are the product of intelligent design, agents that never underwent the million of years of competition with other agents that equipped humans with altruism and general compassion.

But that does not mean that an artificial intelligence won’t have any goals [23][24]. Just that those goals will be simple and their realization remorseless [25].

An artificial general intelligence will do whatever is implied by its initial design. And we will be helpless to stop it from achieving its goals. Goals that won’t automatically respect our values [26].

A likely implication is the total extinction of all of humanity [27].

Further Reading

• What should a reasonable person believe about the Singularity?
• The Singularity: A Philosophical Analysis
• Intelligence Explosion: Evidence and Import
• Why an Intelligence Explosion is Probable
• Artificial Intelligence as a Positive and Negative Factor in Global Risk
• From mostly harmless to civilization-threatening: pathways to dangerous artificial general intelligences
• The Hanson-Yudkowsky AI-Foom Debate
• Facing The Singularity

References

[1] Genetic Algorithms and Evolutionary Computation, talkorigins.org/faqs/genalg/genalg.html
[2] Fixing software bugs in 10 minutes or less using evolutionary computation, genetic-programming.org/hc2009/1-Forrest/Forrest-Presentation.pdf
[3] Automatically Finding Patches Using Genetic Programming, genetic-programming.org/hc2009/1-Forrest/Forrest-Paper-on-Patches.pdf
[4] A Genetic Programming Approach to Automated Software Repair, genetic-programming.org/hc2009/1-Forrest/Forrest-Paper-on-Repair.pdf
[5]GenProg: A Generic Method for Automatic Software Repair, virginia.edu/~weimer/p/weimer-tse2012-genprog.pdf
[6] 29+ Evidences for Macroevolution (The Scientific Case for Common Descent), talkorigins.org/faqs/comdesc/
[7] Thermodynamics, Evolution and Creationism, talkorigins.org/faqs/thermo.html
[8] A Collection of Definitions of Intelligence, vetta.org/documents/A-Collection-of-Definitions-of-Intelligence.pdf
[9] plato.stanford.edu/entries/consciousness/
[10] en.wikipedia.org/wiki/Science
[11] en.wikipedia.org/wiki/Scientific_method
[12] The Automation of Science, sciencemag.org/content/324/5923/85.abstract
[13] Computer Program Self-Discovers Laws of Physics, wired.com/wiredscience/2009/04/newtonai/
[14] List of cognitive biases, en.wikipedia.org/wiki/List_of_cognitive_biases
[15] Intelligence explosion, wiki.lesswrong.com/wiki/Intelligence_explosion
[16] 1% with Neil deGrasse Tyson, youtu.be/9nR9XEqrCvw
[17] Mongol military tactics and organization, en.wikipedia.org/wiki/Mongol_military_tactics_and_organization
[18] Wars of Alexander the Great, en.wikipedia.org/wiki/Wars_of_Alexander_the_Great
[19] Spanish colonization of the Americas, en.wikipedia.org/wiki/Spanish_colonization_of_the_Americas
[20] A Quantitative Test of Hamilton's Rule for the Evolution of Altruism, plosbiology.org/article/info:doi/10.1371/journal.pbio.1000615
[21] Algorithmic information theory, scholarpedia.org/article/Algorithmic_information_theory
[22] Algorithmic probability, scholarpedia.org/article/Algorithmic_probability
[23] The Nature of Self-Improving Artificial Intelligence, selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf
[24] The Basic AI Drives, selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf
[25] Paperclip maximizer, wiki.lesswrong.com/wiki/Paperclip_maximizer
[26] Friendly artificial intelligence, wiki.lesswrong.com/wiki/Friendly_artificial_intelligence
[27] Existential Risk, existential-risk.org

I searched for articles on the topic and couldn't find any.

It seems to me that intelligence explosion makes human annihilation much more likely, since superintelligences will certainly be able to outwit humans, but that a human-level intelligence that could process information much faster than humans would certainly be a large threat itself without any upgrading. It could still discover programmable nanomachines long before humans do, gather enough information to predict how humans will act, etc. We already know that a human-level intelligence can "escape from the box." Not 100% of the time, but a real AI will have the opportunity for many more trials, and its processing abilities should make it far more quick-witted than we are.

I think a non-friendly AI would only need to be 20 years or so more advanced than the rest of humanity to pose a major threat, especially if self-replicating nanomachines are possible. Skeptics of intelligence explosion should still be worried about the creation of computers with unfriendly goal systems. What am I missing?

Nutrition is a case where we have to try to make the best possible use of the data we have no matter how terrible, because we have to eat something now to sustain us while we plan and conduct more experiments.

I want to apply Bayes theorem to make rational health decisions from relatively weak data. I am generally wondering how one can synthesize historical human experiences with incomplete scientific data, in order to make risk-adverse and healthy decisions about human nutrition given limited research.

Example question/hypothesis: Does gluten cause health problems (ie exhibit chronic toxicity) in non-coeliac humans? Is there enough evidence to suggest that avoiding gluten might be a prudent risk-adverse decision for non-coeliacs?

We have some (mostly in vitro) scientific data suggesting that gluten may cause health problems in non-coeliac humans (such as these articles http://evolvify.com/the-case-against-gluten-medical-journal-references/). Let's say for the sake of arguing, that I can somehow convert these studies into a non-unity likelihood ratio for gluten toxicity in humans (although suggestions are welcome here too).

However, we also have prior information that a population of humans has been consuming gluten containing foods for at least 10,000 years, without any blatantly obvious toxic effects. Is there some way to convert this observation (and observations like this) into a prior probability distribution?

If you wanted to check whether your creations were prone to misbehaving, one way would be to leave a big juicy apple hanging tantalisingly from a very visible tree, turn your back for a while, and see what they get up to.

That would work if you were God and your creations were essentially children. But if the creator is us and the creation is a super-intelligent AI, such 'honey-pots' and unlikely to work. They are tricks, and we shouldn't rely on being able to trick a being more intelligent than ourselves - unless its intelligence doesn't matter.

We can use utility indifference to construct such traps. Utility indifference can be informally summarised as:

"There is a stochastic event X which can output 0 or 1. We wish the AI act as if it believed that X=0, with certainty. It's problematic for the AI to have incorrect beliefs, so the result is achieved through the definition of the AI's utility function."

Consider the following situation: a master AI that will create the source code of another, autonomous AI. The master AI is setup to produce the code of a Friendly AI, or a reduced impact AI - but we're not sure if the master AI is trustworthy. It would be good to go through the code very carefully before unleashing it (or erasing it). But if the master AI knows this (and we must assume it does), then it will either obfuscate the dangerous parts or put traps in it, making it risky for us to look too closely.

Then implement the following setup. The code travels through some stochastic event X, such that if X=0, the code is immediately transferred into a disciple AI, completely autonomous, which is at liberty to immediately modify its own code and erase any trace of what it was. If X=1, the code is instead copied to our hard drives for analysis. The AI is given some reason to keep the code short - maybe there is an increasing chance of the code getting erased the longer it is.

On top of this, we implement utility indifference: the master AI will act as if it believed X=0 with certainty, while we have setup X so that X=1 with near-certainty. Then the master AI will be motivated to produce the shortest code compatible with its utility function, in the 'knowledge' that this code will be copied into a powerful autonomous AI, and will never be seen by humans. It has no incentive to trap its code - quite the reverse, as it wants to keep it short.

And therefore we have all the time in the world to go through the code, looking for defection. Other ways of using utility indifference to trap AIs are no doubt possible, but this was the first setup that sprang to my mind.

There's a recent paper(PDF) which finds that people who don't know much about a problem are more inclined to not find out more about that problem. Moreover, the larger scale and more complex a problem looked like, the more likely people were to try to avoid learning more about it, and the more likely they were to trust that pre-existing institutions such as the government could handle the problem. This looks like a potentially interesting form of cognitive bias. It may also explain why people are so unwilling to look at existential risk. There's essentially no issue that occurs on a larger scale than existential risk. This suggests that in trying to get people to understand existential risk, it may make sense to first address the easier to understand existential risks like large asteroids. 

At the FHI, we are currently working on a project around whole brain emulations (WBE), or uploads. One important question is if getting to whole brain emulations first would make subsequent AGI creation

1. more or less likely to happen,
2. more or less likely to be survivable.

If you have any opinions or ideas on this, please submit them here. No need to present an organised overall argument; we'll be doing that. What would help most is any unusual suggestion, that we might not have thought of, for how WBE would affect AGI.

EDIT: Many thanks to everyone who suggested ideas here, they've been taken under consideration.

Underground Q&A session with Nick Bostrom (http://www.nickbostrom.com) on existential risks and artificial intelligence with the Oxford Transhumanists (recorded 10 October 2011).

http://www.youtube.com/watch?v=KQeijCRJSog

The chanses of prevention of the global catastrophe are growing if humans have the goal of it. This is semi-trivial conclusion. But the main question is who should have such goal?

Of course, if we have global government, its main goal should be prevention of global catastrophe. But we do not have global government and most people hate the idea. I find it irrational. But discussion about global goverment is pure theoretical one, because I do not see peaceful ways of its creation. 

If friendly AI take over the world ve will became global government de facto.

Or if imminent global risk will be recognized (asteroid is near), UN could temporaly transform in some kind of global government. 
But some people think that global government itself will be or will soon lead to the global catastrophe - because it could easily implement global measures - and predicate "global" is nessesary to global catastrophes, as I am going to show in one of next posts. For example it could implement global total vaccination that lately will have dangerous consciences.

So we see that idea of global government is very closely connected with idea of global catastrophe. One could lead to another and back.

But as we do not have global government we could only speak about goals of separate people and separate organizations.

People do not have goals. The only think that they have goals, but these are only declarations, which rarely regulate real people's behavior. This is because human beings are not born as rational subjects and their behavior is mostly regulated by unconscious programs known as instincts. 

These instincts are culturally adapted as values. Values are the real reasons of human behavior. "Goals" are what people say about their reasons to others and to themselves.

The problem of how human values influent the course of human history is difficult one. Last year I wrote a book "Futurology. 21 century: immortality or global catastrophe" in Russian together with M.Batin. And the chapter about values was the most difficult one. 

Values are always based on instincts , pleasure and collective behavior (values help to form groups of people who share them). Value is always emotion, it has energy to move a person.

But self-preservation is basic human instinct and so prevention of death and global catastrophe could be human value.

Each value need a group of supporters to exist (value of soccer needs group of fans). Religious values exist only because they have large group of supporters.

In 1960th fight for peace existed and was mass movement. It finally won and lead to limitation of nuclear arsenals in 80th and later. This is good example of how human values prevented global risk without creation of global government.

Now value of "being green" has been created and many people fight CO2.

The problems with such values is that they need very bright picture of risk to attract attention of people. It is not easy to create value to fight global risks in general. But the value of infinite existence of the civilization is much easily imaginable.
So promoting a vision of future galactic super civilization with immortal people could motivate people now to fight global risks in all forms.

Two talks from the Future of Humanity Institute are now online (this is the first time we've done this, so please excuse the lack of polish). The first is Anders Sandberg talking about brain emulations (technical overview), the second is myself talking of the risks of Oracle AIs (informal presentation). They can be found here:

Fesability of whole-brain emulation: http://www.youtube.com/watch?v=3nIzPpF635c&feature=related, initial paper at http://www.philosophy.ox.ac.uk/__data/assets/pdf_file/0019/3853/brain-emulation-roadmap-report.pdf, new paper still to come.

Thinking inside the box: Using and controlling an Oracle AI:http://www.youtube.com/watch?v=Gz9zYQsT-QQ&feature=related, paper at http://www.aleph.se/papers/oracleAI.pdf

(with thanks to Owain Evans)

An ontological crisis happens when an agent's underlying model of reality changes, such as a Newtonian agent realising it was living in a relativistic world all along. These crises are dangerous if they scramble the agent's preferences: in the example above, an agent dedicated to maximise pleasure over time could transition to completely different behaviour when it transitions to relativistic time; depending on the transition, it may react by accelerating happy humans to near light speed, or inversely, ban them from moving - or something considerably more weird.

Peter de Blanc has a sensible approach to minimising the disruption ontological crises can cause to an AI, but this post is concerned with analyzing what happens when such approaches fail. How bad could it be? Well, this is AI, so the default is of course: unbelievably, hideously bad (i.e. situation normal). But in what ways exactly?

In 2005, Hurricane Rita caused 111 deaths.  3 deaths were caused by the hurricane.  90 were caused by the mass evacuation.

The FDA is supposed to approve new drugs and procedures if the expected benefits outweigh the expected costs.  If they actually did this, their errors on both sides (approvals of bad drugs vs. rejections of good drugs) would be roughly equal.  The most-publicized drug withdrawal in the past 10 years was that of Vioxx, which the FDA estimated killed a total of 5165 people over 5 years.  This suggests that the best drug that the FDA rejected during that decade could have saved 1000 people/year.  During that decade, many drugs were (or could have been) approved that might save more than that many lives every year.  Gleevec (invented 1993, approved 2001) is believed to save about 10,000 lives a year.  Herceptin (invented in the 1980s, began human trials 1991, approved for some patients in 1998, more in 2006, and more in 2010) was estimated to save 1,000 lives a year in the United Kingdom, which would translate to 5,000 lives a year in the US.  Patients on Apibaxan (discovered in 2006, not yet approved) have 11% fewer deaths from stroke than patients on warfarin, and stroke causes about 140,000 deaths/year in the US.  To stay below the expected drug-rejection error level of 1000 people/year, given just these three drugs (and assuming that Apibaxan pans out and can save 5,000 lives/year), the FDA would need to have a faulty-rejection rate F such that F(10000) + F(5000) + F(5000) < 1000, F < 5%.  This seems unlikely.

ADDED:  One area where this affects me every day is in branching software repositories.  Every software developer agrees that branching the repository head for test versions and for production versions is good practice.  Yet branching causes, I would estimate, at least half of our problems with test and production releases.  It is common for me to be delayed one to three days while someone figures out that the software isn't running because they issued a patch on one branch and forgot to update the trunk, or forgot to update other development or test versions that are on separate branches.  I don't believe in branching anymore - I think we would have fewer bugs if we just did all development on the trunk, and checked out the code when it worked.  Branching is good for humongous projects where you have public releases that you can't patch on the head, like Firefox or Linux.  But it's out of place for in-house projects where you can just patch the head and re-checkout.  The evidence for this in my personal experience as a software developer is overwhelming; yet whenever I suggest not branching, I'm met with incredulity.

Exercise for the reader:  Find other cases where cautionary measures are <EDIT>taken past the point of marginal utility</EDIT>.

ADDED: I think that this is the problem:  You have observed a distribution of outcome utilities from some category of event followed by you taking some action A.  You observe a new instance of this event.  You want to predict the outcome utility of action A for this event.

Some categories have a power-law outcome distribution with a negative exponent b, indicating there are fewer events of large importance: number of events of size U = e^{c - bU}.  Assume that you don't observe all possible values of U.  Events of importance < U0 are too small to observe; and events with large U are very uncommon.  It is then difficult to tell whether the category has a power-law distribution without a lot of previous observations.

If a lot of event categories have a distribution like this, where big impacts are bad, and they are usually insignificant but sometimes catastrophic, then it's likely rational to treat these events as if they will be catastrophic.  And if you don't have enough observations to know if the distribution is a power-law, or something else, it's rational to treat it as if it were a power-law distribution to be safe.

Could this account for the human risk-aversion "bias"?

If you are the FDA, you are faced with situations where the utility distribution is probably such a power-law distribution mirrored around zero, so there are a few events with very high utility (save lots of lives), and a similar number of events with the negative of that utility (lose that many lives).  I would guess that situations like that are rare in our ancestral environment, though I don't know.

existential-risk.org

^{(Updated 2011-12-16 due to a comment by Nick Bostrom.)}

'Existential Risk FAQ' by Nick Bostrom

(2011) Version 1.0

Short answers to common questions

Link: pdf html

'Existential Risk Prevention as the Most Important Task for Humanity' by Nick Bostrom

(2011) Working paper (revised)

ABSTRACT
Existential risks are those that threaten the entire future of humanity.  Many theories of value imply that even relatively small reductions in net existential risk have enormous expected value.  Despite their importance, issues surrounding human-extinction risks and related hazards remain poorly understood.  In this paper, I clarify the concept of existential risk and develop an improved classification scheme.  I discuss the relation between existential risks and basic issues in axiology, and show how existential risk reduction (via the maxipok rule) can serve as a strongly action-guiding principle for utilitarian concerns.  I also show how the notion of existential risk suggests a new way of thinking about the ideal of sustainability.

Link: pdf html

Sometimes we make a decision in a way which is different to how we think we should make a decision. When this happens, we call it a bias.

When put this way, the first thing that springs to mind is that different people might disagree on whether something is actually a bias. Take the bystander effect. If you're of the opinion that other people are way less important than yourself, then the ability to calmly stand around not doing anything while someone else is in danger would be seen as a good thing. You'd instead be confused by the non-bystander effect, whereby people (when separated from the crowd) irrationally put themselves in danger in order to help complete strangers.

The second thing that springs to mind is that the bias may exist for an evolutionary reason, and not just be due to bad brain architecture. Remember that evolution doesn't always produce the behavior that makes the most intuitive sense. Creatures, including presumably humans, tend to act in a way as to maximize their reproductive success; they don't act in the way that necessarily makes the most intuitive sense.

The statement that humans act in a fitness-maximizing way is controversial. Firstly, we are adapted to our ancestral environment, not our current one. It seems very likely that we're not well adapted to the ready availability of high-calorie food, for example. But this argument doesn't apply to everything. A lot of the biases appear to describe situations which would exist in both the ancestral and modern worlds.

A second argument is that a lot of our behavior is governed by memes these days, not genes. It's certain that the memes that survive are the ones which best reproduce themselves; it's also pretty plausible that exposure to memes can tip us from one fitness-maximizing behavioral strategy to another. But memes forcing us to adopt a highly suboptimal strategy? I'm sceptical. It seems like there would be strong selection pressure against it; to pass the memes on but not let them affect our behavior significantly. Memes existed in our ancestral environments too.

And remember that just because you're behaving in a way that maximizes your expected reproductive fitness, there's no reason to expect you to be consciously aware of this fact.

So let's pretend, for the sake of simplicity, that we're all acting to maximize our expected reproductive success (and all the things that we know lead to it, such as status and signalling and stuff). Which of the biases might be explained away?

The bystander effect

Eliezer points out:

We could be cynical and suggest that people are mostly interested in not being blamed for not helping, rather than having any positive desire to help - that they mainly wish to escape antiheroism and possible retribution.

He lists two problems with this hypothesis. Firstly, that the experimental setup appeared to present a selfish threat to the subjects. This I have no convincing answer to. Perhaps people really are just stupid when it comes to fires, not recognising the risk to themselves, or perhaps this is a gaping hole in my theory.

The other criticism is more interesting. Telling people about the bystander effect makes it less likely to happen? Well, under this hypothesis, of course it would. The key to not being blamed is to formulate a plausible explanation; the explanation "I didn't do anything because no-one else did either" suddenly sounds a lot less plausible when you know about the bystander effect. (And if you know about it, the person you're explaining it to is more likely to as well. We share memes with our friends).

The affect heuristic

This one seems quite complicated and subtle, and I think there may be more than one effect going on here. But one class of positive-affect bias can be essentially described as: phrasing an identical decision in more positive language makes people more likely to choose it. The example given is "saving 150 lives" versus "saving 98% of 150 lives". (OK these aren't quite identical decisions, but the difference in opinion is more than 2% and goes in the wrong direction). Apparently putting in the word 98% makes it sound more positive to most people.

This also seems to make sense if we view it as trying to make a justifiable decision, rather than a correct one. Remember, the 150(ish) lives we're saving aren't our own; there's no selective pressure to make the correct decision, just one that won't land us in trouble.

The key here is that justifying decisions is hard, especially when we might be faced with an opponent more skilled in rhetoric than ourselves. So we are eager for additional rhetoric to be supplied which will help us justify the decision we want to make. If I had to justify saving 150 lives (at some cost), it would honestly never have occurred to me to phrase it as "98% of 153 lives". Even if it had, I'd feel like I was being sneaky and manipulative, and I might accidentally reveal that. But to have the sneaky rhetoric supplied to me by an outside authority, that makes it a lot easier.

This implies a prediction: when asked to justify their decision, people who have succumbed to positive-affect bias will repeat the postive-affective language they have been supplied, possibly verbatim. I'm sure you've met people who quote talking points verbatim from their favorite political TV show; you might assume the TV is doing their thinking for them. I would argue instead that it's doing their justification for them.

Trolley problems

There is a class of people, who I will call non-pushers, who:

• would flick a switch if it would cause a train to run over (and kill) one person instead of five, yet
• would not push a fat man in front of that train (killing him) if it could save the five lives

So what's going on here? Our feeling of shouldness is presumably how social pressure feels from the inside. What we consider right is (unless we've trained ourselves otherwise) likely to be what will get us into the least trouble. So why do non-pushers get into less trouble than pushers, if pushers are better at saving lives?

It seems pretty obvious to me. The pushers might be more altruistic in some vague sense, but they're not the sort of person you'd want to be around. Stand too close to them on a bridge and they might push you off. Better to steer clear. (The people who are tied to the tracks presumably prefer pushers, but they don't get any choice in the matter). This might be what we mean by near and far in this context.

Another way of putting it is that if you start valuing all lives equally, and not put those closest to you first, then you might start defecting in games of reciprocal altruism. Utilitarians appear cold and unfriendly because they're less worried about you and more worried about what's going on in some distant, impoverished nation. They will start to lose the reproductive benefits of reciprocal altruism and socialising.

Global risk

In Cognitive Biases Potentially Affecting Judgment of Global Risks, Eliezer lists a number of biases which could be responsible for people's underestimation of global risks. There seem to be a lot of them. But I think that from an evolutionary perspective, they can all be wrapped up into one.

Group Selection doesn't work. Evolution rewards actions which profit the individual (and its kin) relative to others. Something which benefits the entire group is nice and all that, but it'll increase the frequency of the competitors of your genes as much as it will your own.

It would be all to easy to say that we cannot instinctively understand existential risk because our ancestors have, by definition, never experienced anything like it. But I think that's an over-simplification. Some of our ancestors probably have survived the collapse of societies, but they didn't do it by preventing the society from collapsing. They did it by individually surviving the collapse or by running away.

But if a brave ancestor had saved a society from collapse, wouldn't he (or to some extent, she) become an instant hero with all the reproductive advantage that affords? That would certainly be nice, but I'm not sure the evidence backs it up. Stanislav Petrov was given the cold shoulder. Leading climate scientists are given a rough time, especially when they try and see their beliefs turned into meaningful action. Even Winston Churchill became unpopular after he helped save democratic civilization.

I don't know what the evolutionary reason for hero-indifference would be, but if it's real then it pretty much puts the nail in the coffin for civilization-saving as a reproductive strategy. And that means there's no evolutionary reason to take global risks seriously, or to act on our concerns if we do.

And if we make most of our decisions on instinct - on what feels right - then that's pretty scary.

I was raised to believe that genetically-modified foods are unhealthy to eat and bad for the environment, and given a variety of reasons for this, some of which I now recognize as blatantly false (e.g., human genetic code is isomorphic to fundamental physical law), and a few of which still seem sort of plausible.

Because of this history, I need to anchor my credence heavily downward from my sense of plausibility.

The major reasons I see to believe that GMOs are safe are:

• I would probably think they were dangerous even if they were safe, due to my upbringing.
• In general, whenever someone opposes a particular field of engineering on the grounds that it's unnatural and dangerous, they're usually wrong.
• It's not quite obvious to me that introducing genetically-engineered organisms to a system is significantly more dangerous than introducing non-native naturally-evolved organisms.

The major reason I see to believe that GMOs are dangerous is:

• I might believe they were safe even if they were dangerous, due to "yay science" (which was also part of my upbringing).
• We are designing self-replicating things and using them without reliable containment, thereby effectively releasing them into the wild.

So: green goo, yes or no?

In which I worry that the Less Wrong project might go horribly right. This post belongs to my Altruist Support sequence.

Every project needs a risk assessment.

There's a feeling, just bubbling under the surface here at Less Wrong, that we're just playing at rationality. It's rationality kindergarten. The problem has been expressed in various ways:

• not a whole lot of rationality
• rationalist porn for daydreamers
• not quite as great as everyone seems to think
• shiny distraction
• only good for certain goals

And people are starting to look at fixing it. I'm not worried that their attempts - and mine - will fail. At least we'd have fun and learn something.

I'm worried that they will succeed.

What would such a Super Less Wrong community do? Its members would self-improve to the point where they had a good chance of succeeding at most things they put their mind to. They would recruit new rationalists and then optimize that recruitment process, until the community got big. They would develop methods for rapidly generating, classifying and evaluating ideas, so that the only ideas that got tried would be the best that anyone had come up with so far. The group would structure itself so that people's basic social drives - such as their desire for status - worked in the interests of the group rather than against it.

It would be pretty formidable.

What would the products of such a community be? There would probably be a self-help book that works. There would be an effective, practical guide to setting up effective communities. There would be an intuitive, practical guide to human behavior. There would be books, seminars and classes on how to really achieve your goals - and only the materials which actually got results would be kept. There would be a bunch of stuff on the Dark Arts too, no doubt. Possibly some AI research.

That's a whole lot of material that we wouldn't want to get into the hands of the wrong people.

Dangers include:

• Half-rationalists: people who pick up on enough memes to be really dangerous, but not on enough to realise that what they're doing might be foolish. For example, building an AI without adding the friendliness features.
• Rationalists with bad goals: Someone could rationally set about trying to destroy humanity, just for the lulz.
• Dangerous information discovered: e.g. the rationalist community develops a Theory of Everything that reveals a recipe for a physics disaster (e.g. a cheap way to turn the Earth into a block hole). A non-rationalist decides to exploit this.

If this is a problem we should take seriously, what are some possible strategies for dealing with it?

1. Just go ahead and ignore the issue.
2. The Bayesian Conspiracy: only those who can be trusted are allowed access to the secret knowledge.
3. The Good Word: mix in rationalist ideas with do-good and stay-safe ideas, to the extent that they can't be easily separated. The idea being that anyone who understands rationality will also understand that it must be used for good.
4. Rationality cap: we develop enough rationality to achieve our goals (e.g. friendly AI) but deliberately stop short of developing the ideas too far.
5. Play at rationality: create a community which appears rational enough to distract people who are that way inclined, but which does not dramatically increase their personal effectiveness.
6. Risk management: accept that each new idea has a potential payoff (in terms of helping us avoid existential threats) and a potential cost (in terms of helping "bad rationalists"). Implement the ideas which come out positive.

In the post title, I have suggested an analogy with AI takeoff. That's not entirely fair; there is probably an upper bound to how effective a community of humans can be, at least until brain implants come along. We're probably talking two orders of magnitude rather than ten. But given that humanity already has technology with slight existential threat implications (nuclear weapons, rudimentary AI research), I would be worried about a movement that aims to make all of humanity more effective at everything they do.

Why do we imagine our actions could have consequences for more than a few million years into the future?

Unless what we believe about evolution is wrong, or UFAI is unlikely, or we are very very lucky, we should assume there are already a large number of unfriendly AIs in the universe, and probably in our galaxy; and that they will assimilate us within a few million years.

Therefore, justifications for harming people on Earth today in the name of protecting the entire universe over all time from UFAI in the future, like this one, should not be done.  Our default assumption should be that the offspring of Earth will at best have a short happy life.

ADDED:  If you observe, as many have, that Earth has not yet been assimilated, you can draw one of these conclusions:

1. The odds of intelligent life developing on a planet are precisely balanced with the number of suitable planets in our galaxy, such that after billions of years, there is exactly one such instance.  This is an extremely low-probability argument.  The anthropic argument does not justify this as easily as it justifies observing one low-probability creation of intelligent life.
2. The progression (intelligent life →AI→expansion and assimilation) is unlikely.

Surely, for a Bayesian, the more reasonable conclusion is number 2!  Conclusion 1 has priors we can estimate numerically.  Conclusion 2 has priors we know very little about.

To say, "I am so confident in my beliefs about what a superintelligent AI will do, that I consider it more likely that I live on an astronomically lucky planet, than that those beliefs are wrong", is something I might come up with if asked to draw a caricature of irrationality.

Having all known life on Earth concentrated on a single planet is an existential risk.  So we should try to spread out, right?  As soon as possible?

Yet, if we had advanced civilizations on two planets, that would be two places for unfriendly AI to originate.  If, as many people here believe, a single failed trial ruins the universe, you want to have as few places trying it as possible.  So you don't want any space colonization until after AI is developed.

If we apply that logic to countries, you would want as few industrialized nations as possible until AAI (After AI).  So instead of trying to help Africa, India, China, and the Middle East develop, you should be trying to suppress them.  In fact, if you really believed the calculations I commonly see used in these circles about the probability of unfriendly AI and its consequences, you should be trying to exterminate human life outside of your developed country of choice.  Failing to would be immoral.

And if you apply it within the USA, you need to pick one of MIT and Stanford and Carnegie Mellon, and burn the other two to the ground.

Of course, doing this will slow the development of AI.  But that's a good thing, if UFAI is most likely and has zero utility.

In fact, if slowing development is good, probably the best thing of all is just to destroy civilization and stop development completely.

Do you agree with any of this?  Is there a point where you think it goes too far?  If so, say where it goes too far and explain why.

I see two main flaws in the reasoning.

• Categorization of outcomes as "FAI vs UFAI", with no other possible outcomes recognized, and no gradations within the category of either, and zero utility assigned to UFAI.
• Failing to consider scenarios in which multiple AIs can provide a balance of power.  The purpose of this balance of power may not be to keep humans in charge; it may be to put the AIs in an AI society in which human values will be worthwhile.
• ADDED, after being reminded of this by Vladimir Nesov:  Re. the final point, stopping completely guarantees Earth life will eventually be eliminated; see his comment below for elaboration.

ADDED:  A number of the comments so far imply that the first AI built will necessarily FOOM immediately.  FOOM is an appealing argument.  I've argued in favor of it myself.   But it is not a theorem.  I don't care who you are; you do not know enough about AI and its future development to bet the future of the universe on your intuition that non-FOOMing AI is impossible.  You may even think FOOM is the default case; that does not make it the only case to consider.  In this case, even a 1% chance of a non-foom AI, multiplied by astronomical differences in utility, could justify terrible present disutility.

The most widely appreciated finance theory is the Capital Asset Pricing Model. It basically says that diminishing marginal utility of absolute wealth implies that riskier financial assets should have higher expected returns than less risky assets and that only risk correlated with the market (beta risk) is a whole is important because other risk can be diversified out.

Eric Falkenstein argues that the evidence does not support this theory; that the riskiness of assets (by any reasonable definition) is not positively correlated with return (some caveats apply). He has a paper (long but many parts are skimmable; not peer reviewed; also on SSRN) as well as a book on the topic. I recommend reading parts of the paper.

The gist of his competing theory is that people care mostly about relative gains rather than absolute gains. This implies that riskier financial assets will not have higher expected returns than less risky assets. People will not require a higher return to hold assets with higher undiversifiable variance because everyone is exposed to the same variance and people only care about their relative wealth.

Falkenstein has a substantial quantity of evidence to back up his claim. I am not sure if his competing theory is correct, but I find the evidence against the standard theory quite convincing.

If risk is not correlated with returns, then anyone who is mostly concerned with absolute wealth can profit from this by choosing a low beta risk portfolio.

This topic seems more appropriate for the discussion section, but I am not completely sure, so if people think it belongs in the main area, let me know.

Added some (hopefully) clarifying material:

All this assumes that you eliminate idiosyncratic risk through diversification. Technically impossible, but you can get it reasonably low. The R's are all *instantaneous* returns; though since these are linear models they apply to geometrically accumulated returns as well. The idea that E(R_asset) are independent of past returns is a background assumption for both models and most of finance.

Beta_portfolio = Cov(R_portfolio, R_market)/variance(R_market)

In CAPM your expected and variance are:

E(R_portfolio) = R_rfree + Beta_portfolio * (E(R_market) - R_rfree)
Var(R_portfolio) = Beta_portfolio * Var(R_market)

in Falkenstein's model your expected return are:

E(R_portfolio) = R_market       # you could also say = R_rfree; the point is that its a constant
Var(R_portfolio) = Beta_portfolio * Var(R_market)

The major caveat being that it doesn't apply very close to Beta_portfolio = 0; Falkenstein attributes this to liquidity benefits. And it doesn't apply to very high Beta_portfolio; he attributes this to "buying hope". See the paper for more.

Falkenstein argues that his model fits the facts more closely than CAPM. Assuming Falkenstein's model describes reality, if your utility declines with rising Var(R_portfolio) (the standard assumption), then you'll want to hold a portfolio with a beta of zero; or taking into account the caveats, a low Beta_portfolio. If your utility is declining with Var(R_portfolio - R_market), then you'll want to hold the market portfolio. Both of these results are unambiguous since there's no trade off between either measure of risk and return.

Some additional evidence from another source, and discussion: http://falkenblog.blogspot.com/2010/12/frazzini-and-pedersen-simulate-beta.html

I would like to draw attention to the honours thesis of Katja Grace (Meteuphoric).

Link: meteuphoric.wordpress.com/2010/11/02/anthropic-principles-agree-on-bigger-future-filters/
PDF: dl.dropbox.com/u/6355797/Anthropic%20Reasoning%20in%20the%20Great%20Filter.pdf

My main point was that two popular anthropic reasoning principles, the Self Indication Assumption (SIA) and the Self Sampling Assumption (SSA), as well as Full Non-indexical Conditioning (FNC)  basically agree that future filter steps will be larger than we otherwise think, including the many future filter steps that are existential risks.

What do you think? (Consider commenting over on her blog, Robin Hanson is also there.)