(Cross-posted from Rational Altruist. See also recent posts on time-discounting and self-driving cars.)

When talking about the future, I often encounter two (quite different) stories describing why the future might be good:

1. Decisions will be made by people whose lives are morally valuable and who want the best for themselves. They will bargain amongst each other and create a world that is good to live in. Because my values are roughly aligned with their aggregate preferences, I expect them to create a rich and valuable world (by my lights as well as theirs).
2. Some people in the future will have altruistic values broadly similar to my own, and will use their influence to create a rich and valuable world (by my lights as well as theirs).

Which of these pictures we take more seriously has implications for what we should do today. I often have object level disagreements which seem to boil down to disagreement about which of these pictures is more important, but rarely do I see serious discussion of that question. (When there is discussion, it seems to turn into a contest of political ideologies rather than facts.)

If we take picture (1) seriously, we may be interested in ensuring that society continues to function smoothly, that people are aware of and pursue what really makes them happy, that governments are effective, markets are efficient, externalities are successfully managed, etc. If we take picture (2) seriously, we are more likely to be concerned with changing what the people of the future value, bolstering the influence of people who share our values, and ensuring that altruists are equipped to embark on their projects successfully.

I'm mostly concerned with the very long run---I am wondering what conditions will prevail for most of the people who live in the future, and I expect most of them to be alive very far from now.

It seems to me that there are two major factors that control the relative importance of pictures (1) and (2): how prominent should we expect altruism to be in the future, and how efficiently are altruistic vs. selfish resources being used to create value? My answer to the second question is mostly vague hand-waving, but I think I have something interesting to say on the first question.

How much altruism do we expect?

I often hear people talking about the future, and the present for that matter, as if we are falling towards a Darwinian attractor of cutthroat competition and vanishing empathy (at least as a default presumption, which might be averted by an extraordinary effort). I think this picture is essentially mistaken, and my median expectation is that the future is much more altruistic than the present.

Dose natural selection select for self-interest?

In the world of today, it may seem that humans are essentially driven by self-interest, that this self-interest was a necessary product of evolution, that good deeds are principally pursued instrumentally in service of self-interest, and that altruism only exists at all because it is too hard for humans to maintain a believable sociopathic facade.

If we take this situation and project it towards a future in which evolution has had more time to run its course, creating automations and organizations less and less constrained by folk morality, we may anticipate an outcome in which natural selection has stripped away all empathy in favor of self-interest and effective manipulation. Some may view this outcome as unfortunate but inevitable, others may view it as a catastrophe which we should work to avert, and still others might view it as a positive outcome in which individuals are free to bargain amongst themselves and create a world which serves their collective interest.

But evolution itself does not actually seem to favor self-interest at all. No matter what your values, if you care about the future you are incentivized to survive, to acquire resources for yourself and your descendants, to defend yourself from predation, etc. etc. If I care about filling the universe with happy people and you care about filling the universe with copies of yourself, I'm not going to set out by trying to make people happy while allowing you and your descendants to expand throughout the universe unchecked. Instead, I will pursue a similar strategy of resource acquisition (or coordinate with others to stop your expansion), to ensure that I maintain a reasonable share of the available resources which I can eventually spend to help shape a world I consider value. (See here for a similar discussion.)

This doesn't seem to match up with what we've seen historically, so if I claim that it's relevant to the future I have some explaining to do.

Historical distortions

Short-range consequentialism

One reason we haven't seen this phenomenon historically is that animals don't actually make decisions by backwards-chaining from a desired outcome. When animals (including humans) engage in goal-oriented behavior, it tends to be pretty local, without concern for consequences which are distant in time or space. To the extent that animal behavior is goal-oriented at a large scale, those goals are largely an emergent property of an interacting network of drives, heuristics, etc. So we should expect animals to have goals which lead them to multiply and acquire resources, even when those drives are pursued short-sightedly. And indeed, that's what we see. But it's not the fault of evolution alone---it is a product of evolution given nature's inability to create consequentialist reasoners.

Casually, we seem to observe a similar situation with respect to human organizations---organizations which value expansion for its own sake (or one of its immediate consequences) are able to expand aggressively, while organizations which don't value expansion have a much harder time deciding to expand for instrumental reasons without compromising their values.

Hopefully, this situation is exceptional in history. If humans ever manage to build systems which are properly consequentialist---organizations or automations which are capable of expanding because it is instrumentally useful---we should not expect natural selection to discriminate at all on the basis of those systems' values.

Value drift

Human's values are also distorted by the process of reproduction. A perfect consequentialist would prefer to have descendants who share their values. (Even if I value diversity or freedom of choice, I would like my children to at least share those values, at least if I want that freedom and diversity to last more than one generation!) But humans don't have this option---the only way we can expand our influence is by creating very lossy copies. And so each generation is populated by a fresh batch of humans with a fresh set of values, and the values of our ancestors only have an extremely indirect effect on the world of today.

Again, a similar problem afflicts human organizations. If I create a foundation that I would like to persist for generations, the only way it can expand its influence is by hiring new staff. And since those staff have a strong influence over what my foundation will do, the implicit values of my foundation will slowly but surely be pulled back to the values of the pool of human employees that I have to draw from.

These constraints distort evolution, causing selection to act only those traits which can be reliably passed on from one generation to the next. In particular, this exacerbates the problem from the preceding section---even to the extent that humans can engage in goal-oriented reasoning and expand their own influence instrumentally, these tendencies can not be very well encoded in genes or passed on to the next generation in other ways. This is perhaps the most fundamental change which would result from the development of machine intelligences. If it were possible to directly control the characteristics and values of the next generation, evolution would be able to act on those characteristics and values directly.

So what does natural selection select for?

If the next generation is created by the current generation, guided by the current generation's values, then the properties of the next generation will be disproportionately affected by those who care most strongly about the future.

In finance: if investors have different time preferences, those who are more patient will make higher returns and eventually accumulate much wealth. In demographics: if some people care more about the future, they may have more kids as a way to influence it, and therefore be overrepresented in future generations. In government: if some people care about what government looks like in 100 years, they will use their political influence to shape what the government looks like in 100 years rather than trying to win victories today.

What natural selection selects for is patience. In a thousand years, given efficient natural selection, the most influential people will be those who today cared what happens in a thousand years. Preferences about what happens to me (at least for a narrow conception of personal identity) will eventually die off, dominated by preferences about what society looks like on the longest timescales.

I think this picture is reasonably robust. There are ways that natural selection (/ efficient markets) can be frustrated, and I would not be too surprised if these frustrations persisted indefinitely, but nevertheless this dynamic seems like one of the most solid features of an uncertain future.

What values are we starting with?

Most of people's preferences today seem to concern what happens to them in the near term. If we take the above picture seriously, these values will eventually have little influence over society. Then the question becomes: if we focus only on humanity's collective preferences over the long term, what do those preferences look like? (Trying to characterize preferences as "altruistic" or not no longer seems useful as we zoom in.)

This is an empirical question, which I am not very well-equipped to value. But I can make a few observations that ring true to me (though my data is mostly drawn from academics and intellectuals, who may fail to be representative of normal people in important ways even after conditioning on the "forward-looking" part of people's values):

1. When people think about the far future (and thus when they articulate their preferences for the far future) they seem to engage a different mode of reasoning, more strongly optimized to produce socially praise-worthy (and thus prosocial) judgments. This might be characterized as a bias, but to the extent we can talk about human preferences at all they seem to be a result of these kinds of processes (and to the extent that I am using my own altruistic values to judge futures, they are produced by a similar process). This effect seems to persist even when we are not directly accountable for our actions.
2. People mostly endorse their own enlightened preferences, and look discouragingly at attempts to lock-in hastily considered values (though they often seem to have overconfident views about what their enlightened preferences will look like, which admittedly might interfere with their attempts at reflection).
3. I find myself sympathetic to very many people's accounts of their own preferences about the future, even where those accounts different significantly from my own. I would be surprised if the distribution of moral preferences was too scattered.
4. To the extent that people care especially about their species, their nation, their family, themselves, etc. : they seem to be sensitive to fairness considerations (and rarely wish e.g. to spend a significant fraction of civilization's resources on themselves), their preferences seem to be only a modest distortion of aggregative values (wanting people with property X to flourish is not so different from wanting people to flourish, if property X is some random characteristic without moral significance), and human preferences seem to somewhat reliably drift in the direction of more universal concern as basic needs are addressed and more considerations are considered.

After cutting away all near-term interests, I expect that contemporary human society's collective preferences are similar to their stated moral preferences, with significant disagreement on many moral judgments. However, I expect that these values support reflection, that upon reflection the distribution of values is not too broad, and that for the most part these values are reasonably well-aligned. With successful bargaining, I expect a mixture of humanity's long-term interests to be only modestly (perhaps a factor of 10, probably not a factor of 1000) worse than my own values (as judged by my own values).

Moreover, I have strong intuitions to emphasize those parts of my values which are least historically contingent. (I accept that all of my values are contingent, but am happier to accept those values that are contingent on my biological identity than those that are contingent on my experiences as a child, and happier to accept those that are contingent on my experiences as a child than those that are contingent on my current blood sugar.) And I have strong reciprocity intuitions that exacerbate this effect and lead me to be more supportive of my peers' values. These effects make me more optimistic about a world determined by humanity's aggregate preferences than I otherwise would be.

How important is altruism?

(The answer to this question, unlike the first one, depends on your values: how important to what? I will answer from my own perspective. I have roughly aggregative values, and think that the goodness of a world with twice as many happy people is twice as high.)

Even if we know a society's collective preferences, it is not obvious what their relative importance is. At what level of prevalence would the contributions of explicit altruism become the source of value? If altruists are 10% of the influence-weighted population, do the contributions of the altruists matter? What if altruists are 1% of the population? A priori, it seems clear that the explicit altruists should do at least as much good--on the altruistic account--as any other population (otherwise they could decide to jump ship and become objectivists, or whatever). But beyond that, it isn't clear that altruists should create much more value--even on the altruistic account--than people with other values.

I suspect that explicit altruistic preferences create many times more value than self-interest or other nearly orthogonal preferences. So in addition to expecting a future in which altruistic preferences play a very large role, I think that altruistic preferences would be responsible for most of the value even if they controlled only 1% of the resources.

One significant issue is population growth. Self-interest may lead people to create a world which is good for themselves, but it is unlikely to inspire people to create as many new people as they could, or use resources efficiently to support future generations. But it seems to me that the existence of large populations is a huge source of value. A barren universe is not a happy universe.

A second issue is that population characteristics may also be an important factor in the goodness of the world, and self-interest is unlikely to lead people to ensure that each new generation has the sorts of characteristics which would cause them to lead happy lives. It may happen by good fortune that the future is full of people who are well-positioned to live rich lives, but I don't see any particular reason this would happen. Instead, we might have a future "population" in which almost all resources support automation that doesn't experience anything, or a world full of minds which crave survival but experience no joy, or etc.; "self-interest" wouldn't lead any of these populations to change themselves to experience more happiness. It's not clear why we would avoid these outcomes except by a law of nature that said that productive people were happy people (which seems implausible to me) or by coordinating to avoid these outcomes.

(If you have different values, such that there is a law [or at least guideline] of nature: "productive people are morally valuable people," then this analysis may not apply to you. I know several such people, but I have a hard time sympathizing with their ethics.)

Conclusion

I think that the goodness of a world is mostly driven by the amount of explicit optimization that is going on to try and make the world good (this is all relative to my values, though a similar analysis seems to carry with respect to other aggregative values). This seems to be true even if relatively little optimization is going on. Fortunately, I also think that the future will be characterized by much higher influence for altruistic values. If I thought altruism was unlikely to win out, I would be concerned with changing that. As it is, I am instead more concerned with ensuring that the future proceeds without disruptions. (Though I still think it is worth it to try and increase the prevalence of altruism faster, most of all because this seems like a good approach to minimizing the probability of undesired disruptions.)