Non-political follow-up to: Ungrateful Hitchhikers (offsite)

Related to: Prices or Bindings?, The True Prisoner's Dilemma

Summary: Situations like the Parfit's Hitchhiker problem select for a certain kind of mind: specifically, one that recognizes that an action can be optimal, in a self-interested sense, even if it can no longer cause any future benefit.  A mind that can identify such actions might put them in a different category which enables it to perform them, in defiance of the (futureward) consequentialist concerns that normally need to motivate it.  Our evolutionary history has put us through such "Parfitian filters", and the corresponding actions, viewed from the inside, feel like "something we should do", even if we don’t do it, and even if we recognize the lack of a future benefit.  Therein lies the origin of our moral intuitions, as well as the basis for creating the category "morality" in the first place.

Introduction: What kind of mind survives Parfit's Dilemma?

Parfit's Dilemma – my version – goes like this: You are lost in the desert and near death.  A superbeing known as Omega finds you and considers whether to take you back to civilization and stabilize you.  It is a perfect predictor of what you will do, and only plans to rescue you if it predicts that you will, upon recovering, give it $0.01 from your bank account.  If it doesn’t predict you’ll pay, you’re left in the desert to die. [1]

So what kind of mind wakes up from this?  One that would give Omega the money.  Most importantly, the mind is not convinced to withhold payment on the basis that the benefit was received only in the past.  Even if it recognizes that no future benefit will result from this decision -- and only future costs will result -- it decides to make the payment anyway.

The self-indication assumption (SIA) states that

Given the fact that you exist, you should (other things equal) favor hypotheses according to which many observers exist over hypotheses on which few observers exist.

The reason this is a bad assumption might not be obvious at first.  In fact, I think it's very easy to miss.

Argument for SIA posted on Less Wrong

First, let's take a look at a argument for SIA that appeared at Less Wrong (link).  Two situations are considered.

1.  we imagine that there are 99 people in rooms that have a blue door on the outside (1 person per room).  One person is in a room with a red door on the outside.  It was argued that you are in a blue door room with probability 0.99.

2.  Same situation as above, but first a coin is flipped.  If heads, the red door person is never created.  If tails, the blue door people are never created.  You wake up in a room and know these facts.  It was argued that you are in a blue door room with probability 0.99.

So why is 1. correct and 2. incorrect?  The first thing we have to be careful about is not treating yourself as special.  The fact that you woke up just tells you that at least one conscious observer exists. 

In scenario 1 we basically just need to know what proportion of conscious observers are in a blue door room.  The answer is 0.99.

In scenario 2 you never would have woken up in a room if you hadn't been created.  Thus, the fact that you exist is something we have to take into account.  We don't want to estimate P(randomly selected person, regardless of if they exist or not, is in a blue door room).  That would be ignoring the fact that you exist.  Instead, the fact that you exist tells us that at least one conscious observer exists.  Again, we want to know what proportion of conscious observers are in blue door rooms.  Well, there is a 50% chance (if heads landed) that all conscious observers are in blue door rooms, and a 50% chance that all conscious observers are in red door rooms.  Thus, the marginal probability of a conscious observer being in a blue door room is 0.5.

The flaw in the more detailed Less Wrong proof (see the post) is when they go from step C to step D.  The *you* being referred to in step A might not exist to be asked the question in step D.  You have to take that into account.

General argument for SIA and why it's wrong

Let's consider the assumption more formally.

Assume that the number of people to be created, N, is a random draw from a discrete uniform distribution¹ on {1,2,...,N_max}.  Thus, P(N=k)=1/N_max, for k=1,...,N_max.  Assume N_max is large enough so that we can effectively ignore finite sample issues (this is just for simplicity).

Assume M= N_max*(N_max+1)/2 possible people exist, and we arbitrarily label them 1,...,M.  After the size of the world, say N=n, is determined, then we randomly draw n people from the M possible people.

After the data are collected we find out that person x exists.

We can apply Bayes' theorem to get the posterior probability:

P(N=k|x exists)=k/M, for k=1,...,N_max.

The prior probability was uniform, but the posterior favors larger worlds.  QED.

Well, not really.

The flaw here is that we conditioned on person x existing, but person x only became of interest after we saw that they existed (peeked at the data).

What we really know is that at least one conscious observer exists -- there is nothing special about person x.

So, the correct conditional probability is:

P(N=k|someone exists)=1/N_max, for k=1,...,N_max.

Thus, prior=posterior and SIA is wrong.

Egotism

The flaw with SIA that I highlighted here is it treats you as special, as if you were labeled ahead of time.  But the reality is, no matter who was selected, they would think they are the special person.  "But I exist, I'm not just some arbitrary person.  That couldn't happen in small world.  It's too unlikely."  In reality, that fact that I exist just means someone exists. I only became special after I already existed (peeked at the data and used it to construct the conditional probability).

Here's another way to look at it.  Imagine that a random number between 1 and 1 trillion was drawn.  Suppose 34,441 was selected.  If someone then asked what the probability of selecting that number was, the correct answer is 1 in 1 trillion.  They could then argue, "that's too unlikely of an event.  It couldn't have happened by chance."  However, because they didn't identify the number(s) of interest ahead of time, all we really can conclude is that a number was drawn, and drawing a number was a probability 1 event.

I give more examples of this here.

I think Nick Bostrom is getting at the same thing in his book (page 125):

..your own existence is not in general a ground for thinking that hypotheses are more likely to be true just by virtue of implying that there is a greater total number of observers. The datum of your existence tends to disconfirm hypotheses on which it would be unlikely that any observers (in your reference class) should exist; but that’s as far as it goes. The reason for this is that the sample at hand—you—should not be thought of as randomly selected from the class of all possible observers but only from a class of observers who will actually have existed. It is, so to speak, not a coincidence that the sample you are considering is one that actually exists. Rather, that’s a logical consequence of the fact that only actual observers actually view themselves as samples from anything at all

Related arguments are made in this LessWrong post.  

¹ for simplicity I'm assuming a uniform prior... the prior isn't the issue here

But I hope that our Mars probes will discover nothing. It would be good news if we find Mars to be completely sterile. Dead rocks and lifeless sands would lift my spirit.

Conversely, if we discovered traces of some simple extinct life form—some bacteria, some algae—it would be bad news. If we found fossils of something more advanced, perhaps something looking like the remnants of a trilobite or even the skeleton of a small mammal, it would be very bad news. The more complex the life we found, the more depressing the news of its existence would be. Scientifically interesting, certainly, but a bad omen for the future of the human race.

— Nick Bostrom, in Where Are They? Why I hope that the search for extraterrestrial life finds nothing

This post is a reply to Robin Hanson's recent OB post Very Bad News, as well as Nick Bostrom's 2008 paper quoted above, and assumes familiarity with Robin's Great Filter idea. (Robin's server for the Great Filter paper seems to be experiencing some kind of error. See here for a mirror.)

Suppose Omega appears and says to you:

(Scenario 1) I'm going to apply a great filter to humanity. You get to choose whether the filter is applied one minute from now, or in five years. When the designated time arrives, I'll throw a fair coin, and wipe out humanity if it lands heads. And oh, it's not the current you that gets to decide, but the version of you 4 years and 364 days from now. I'll predict his or her decision and act accordingly.

I hope it's not controversial that the current you should prefer a late filter, since (with probability .5) that gives you and everyone else five more years of life. What about the future version of you? Well, if he or she decides on the early filter, that would constitutes a time inconsistency. And for those who believe in multiverse/many-worlds theories, choosing the early filter shortens the lives of everyone in half of all universes/branches where a copy of you is making this decision, which doesn't seem like a good thing. It seems clear that, ignoring human deviations from ideal rationality, the right decision of the future you is to choose the late filter.

Katja Grace has just presented an ingenious model, claiming that SIA combined with the great filter generates its own variant of the doomsday argument. Robin echoed this on Overcoming Bias. We met soon after Katja had come up with the model, and I signed up to it, saying that I could see no flaw in the argument.

Unfortunately, I erred. The argument does not work in the form presented.

First of all, there is the issue of time dependence. We are not just a human level civilization drifting through the void in blissful ignorance about our position in the universe. We know (approximately) the age of our galaxy, and the time elapsed since the big bang.

How is this relevant? It is relevant because all arguments about the great filter are time-dependent. Imagine we had just reached consciousness and human-level civilization, by some fluke, two thousand years after the creation of our galaxy, by an evolutionary process that took two thousand years. We see no aliens around us. In this situation, we have no reason to suspect any great filter; if we asked ourselves "are we likely to be the first civilization to reach this stage?" then the answer is probably yes. No evidence for a filter.

Imagine, instead, that we had reached consciousness a trillion years into the life of our galaxy, again via an evolutionary process that took two thousand years, and we see no aliens or traces of aliens. Then the evidence for a filter is overwhelming; something must have stopped all those previous likely civilizations from emerging into the galactic plane.

So neither of these civilizations can be included in our reference class (indeed, the second one can only exist if we ourselves are filtered!). So the correct reference class to use is not "the class of all potential civilizations in our galaxy that have reached our level of technological advancement and seen no aliens", but "the class of all potential civilizations in our galaxy that have reached our level of technological advancement at around the same time as us and seen no aliens". Indeed, SIA, once we update on the present, cannot tell us anything about the future.

But there's more.

If our civilization doesn’t collapse then in 50 to 1000 years humanity will almost certainly start colonizing the galaxy.  This seems inconsistent with the fact that there are a huge number of other planets in our galaxy but we have not yet found evidence of extraterrestrial life.  Drawing from this paradox, Robin Hanson writes that there should be some great filter “between death and expanding lasting life, and humanity faces the ominous question: how far along this filter are we?”

Katja Grace reasons that this filter probably lies in our future and so we are likely doomed.   (Hanson agrees with Grace’s argument.)  Please read Grace’s post before reading the rest of this post.

Small groups of humans have been in situations similar to that currently faced by our entire species.  To see this imagine you live in a prehistoric hunter gatherer tribe of 200 people.  Your tribe lives on a large, mostly uninhabited island.  Your grandparents, along with a few other people came over to the island about 30 years ago.  Since arriving on the island no one in your tribe has encountered any other humans.   You figure that if your civilization doesn’t get wiped out then in 100 or so years your people will multiply in number and spread throughout the island.  If your civilization does so spread, then any new immigrants to the island would quickly encounter your tribe.

Why, you wonder, have you not seen other groups of humans on your island?  You figure that it’s either because your tribe was the first to arrive on the island, or because your island is prone to extinction disasters that periodically wipes out all its human inhabitance.  Using anthropic reasoning similar to that used by Katja Grace you postulate the existence of four types of islands:

1)  Islands that are easy to reach and are not prone to disaster.
2)  Islands that are hard to reach and are not prone to disaster.
3)  Islands that are easy to reach and are prone to disaster.
4)  Islands that are hard to reach and are prone to disaster.

You conclude that your island almost certainly isn’t type (1) because if it were you would have almost certainly seen other humans on the island.   You also figure that on type (3) islands lots of civilizations will start and then be destroyed.  Consequently, most of the civilizations that find themselves on types (2), (3) or (4) islands are in fact on type (3) islands.  Your tribe, you therefore reason, will probably be wiped out by a disaster.

I believe that the argument for why you are probably on a type (3) island is analogous to that of why the great filter probably lies in our future because both come about from updating “on your own existence by weighting possible worlds as more likely the more observers they contain.”

If, therefore, Grace’s anthropic reasoning is correct then most of the time when a prehistoric group of humans settled a large, uninhabited island, that group went extinct.

It has been well over a year since I first read Permutation City and relating writings on the internet on Greg Egan's dust theory. It still haunts me. The theory has been discussed tangentially in this community, but I haven't found an article that directly addresses the rationality of Egan's own dismissal of the theory.

In the FAQ, Egan says things like:

I wrote the ending as a way of dramatising[sic] a dissatisfaction I had with the “pure” Dust Theory that I never could (and still haven't) made precise (see Q5): the universe we live in is more coherent than the Dust Theory demands, so there must be something else going on.

and:

I have yet to hear a convincing refutation of it on purely logical grounds...

However, I think the universe we live in provides strong empirical evidence against the “pure” Dust Theory, because it is far too orderly and obeys far simpler and more homogeneous physical laws than it would need to, merely in order to contain observers with an enduring sense of their own existence. If every arrangement of the dust that contained such observers was realised, then there would be billions of times more arrangements in which the observers were surrounded by chaotic events, than arrangements in which there were uniform physical laws.