## Simplified Anthropic Doomsday

1 02 September 2017 08:37PM

Here is a simplified version of the Doomsday argument in Anthropic decision theory, to get easier intuitions.

Assume a single agent A exists, an average utilitarian, with utility linear in money. Their species survives with 50% probability; denote this event by S. If the species survives, there will be 100 people total; otherwise the average utilitarian is the only one of its kind. An independent coin lands heads with 50% probability; denote this event by H.

Agent A must price a coupon CS that pays out €1 on S, and a coupon CH that pays out €1 on H. The coupon CS pays out only on S, thus the reward only exists in a world where there are a hundred people, thus if S happens, the coupon CS is worth (€1)/100. Hence its expected worth is (€1)/200=(€2)/400.

But H is independent of S, so (H,S) and (H,¬S) both have probability 25%. In (H,S), there are a hundred people, so CH is worth (€1)/100. In (H,¬S), there is one person, so CH is worth (€1)/1=€1. Thus the expected value of CH is (€1)/4+(€1)/400 = (€101)/400. This is more than 50 times the value of CS.

Note that C¬S, the coupon that pays out on doom, has an even higher expected value of (€1)/2=(€200)/400.

So, H and S have identical probability, but A assigns CS and CH different expected utilities, with a higher value to CH, simply because S is correlated with survival and H is independent of it (and A assigns an ever higher value to C¬S, which is anti-correlated with survival). This is a phrasing of the Doomsday Argument in ADT.

## The Doomsday argument in anthropic decision theory

5 31 August 2017 01:44PM

EDIT: added a simplified version here.

Crossposted at the intelligent agents forum.

In Anthropic Decision Theory (ADT), behaviours that resemble the Self Sampling Assumption (SSA) derive from average utilitarian preferences (and from certain specific selfish preferences).

However, SSA implies the doomsday argument, and, to date, I hadn't found a good way to express the doomsday argument within ADT.

This post will remedy that hole, by showing how there is a natural doomsday-like behaviour for average utilitarian agents within ADT.

## Sleeping Beauty Problem Can Be Explained by Perspective Disagreement (II)

4 19 July 2017 11:00PM

This is the second part of my argument. It mainly involves a counter example to SIA and Thirdism.

Different part of the argument can be found here: IIIIIIIV

The 81-Day Experiment(81D):

There is a circular corridor connected to 81 rooms with identical doors. At the beginning all rooms have blue walls. A random number R is generated between 1 and 81. Then a painter randomly selects R rooms and paint them red. Beauty would be put into a drug induced sleep lasting 81 day, spending one day in each room. An experimenter would wake her up if the room she currently sleeps in is red and let her sleep through the day if the room is blue. Her memory of each awakening would be wiped at the end of the day. Each time after beauty wakes up she is allowed to exit her room and open some other doors in the corridor to check the colour of those rooms. Now suppose one day after opening 8 random doors she sees 2 red rooms and 6 blue rooms. How should beauty estimate the total number of red rooms(R).

For halfers, waking up in a red room does not give beauty any more information except that R>0. Randomly opening 8 doors means she took a simple random sample of size 8 from a population of 80. In the sample 2 rooms (1/4) are red. Therefore the total number of red rooms(R) can be easily estimated as 1/4 of the 80 rooms plus her own room, 21 in total.

For thirders, beauty's own room is treated differently.As SIA states, finding herself awake is  as if she chose a random room from the 81 rooms and find out it is red. Therefore her room and the other 8 rooms she checked are all in the same sample. This means she has a simple random sample of size 9 from a population of 81. 3 out of 9 rooms in the sample (1/3) are red. The total number of red rooms can be easily estimated as a third of the 81 rooms, 27 in total.

If a bayesian analysis is performed R=21 and R=27 would also be the case with highest credence according to halfers and thirders respectively. It is worth mentioning if an outside Selector randomly chooses 9 rooms and check them, and it just so happens those 9 are the same 9 rooms beauty saw (her own room plus the 8 randomly chosen rooms), the Selector would estimate R=27 and has the highest credence for R=27. Because he and the beauty has the exact same information about the rooms their answer would not change even if they are allowed to communicate. So again, there will be a perspective disagreement according to halfers but not according to thirders. Same as mentioned in part I.

However, thirder's estimation is very problematic. Because beauty believes the 9 rooms she knows is a fair sample of all 81 rooms (since she used it in statistical estimation), it means red rooms (and blue rooms) are not systematically over- or under-represented. Since beauty is always going to wake up in a red room, she has to conclude the other 8 rooms is not a fair sample. Red rooms have to be systematically underrepresent in those 8 rooms. This means even before beauty decides which doors she wants to open we can already predict with certain confidence that those 8 rooms is going to contains less reds than the average of the 80 suggests. This supernatural predicting power is a strong evidence against SIA and thirding.

The argument can also be structured this way. Consider the following three statements:

A: The 9 rooms is an unbiased sample of the 81 rooms.

B: Beauty is guaranteed to wake up in a red room

C: The 8 rooms beauty choose is an unbiased sample of the other 80 rooms.

These statements cannot be all true at the same time. Thirders accept A and B meaning they must reject C. In fact they must conclude the 8 rooms she choose would be biased towards blue. This contradicts the fact that the 8 rooms are randomly chosen.

(EDIT Aug 1. I think the best answer thirders shall give is accept C since it is obviously a simple random sample. Adding another red room to this will make the 9 rooms biased from his perspective. However they can argue if a selector saw the same 9 rooms through random selection then it is unbiased from the selector's perspective. Thirders could argue she must answer from the selector's perspective instead of her own. Main reason being she is undergoing potential memory wipes so her perspective is somewhat "compromised". However, with this explanation thirder must confirm the perspective disagreement between beauty and the selector about whether or not the 9 rooms are biased. It also utilize perspective reasoning followed. In another word perspective disagreement is not unique to halfers and shall not be treated as a weakness.)

It is also easy to see why beauty should not estimate R the same way as the selector does. There are about 260 billion distinct combinations to pick 9 rooms out of 81. The selector has a equal chance to see any one of those 260 billion combinations. Beauty on the other hand could only possibility see a subset of the combinations. If a combination does not contains a red room, beauty would never see it. Furthermore, the more red rooms a combination contains the more awakening it has leading to a greater chance for a beauty to select the said combination. Therefore while the same 9 rooms is a unbiased sample for the selector it is a sample biased towards red for beauty.

(EDIT Aug 1. We can show this another way. Let the selector, halfer beauty and thirder beauty do a large number repeated estimation on the same set of rooms. The selector and halfer's estimations would be concentrated around the true value of R, where as thirders answer would be concentrated on some value larger.)

One might want to argue after the selector learns a beauty has the knowledge of the same 9 rooms he should lower his estimation of R to the same as beauty’s. After all beauty could only know combinations in a subset biased towards red. The selector should also reason his sample is biased towards red. This argument is especially tempting for SSA supporters since if true it means their answer also yields no disagreements. Sadly this notion is wrong, the selector ought to remain his initial estimation. To the selector a beauty knowing the same 9 rooms simply means after waking up in one of the red rooms in his sample, beauty made a particular set of random choices coinciding said sample. It offers him no new information about the other rooms. This point can be made clearer if we look at how people reach to an agreement in an ordinary problem. Which would be shown by another thought experiment in the next part.

Part III can be found at here

## Sleeping Beauty Problem Can Be Explained by Perspective Disagreement (I)

1 19 July 2017 10:11PM

EDIT: It seems this is not getting much attention except for part II with the counter example to SIA. Let me try to intrigue you in to reading this. It provides a systematic argument for double halving in the sleeping beauty problem, if my reasoning is correct and perspective disagreement is indeed the key, we can rejects Doomsday Argument, and disagree with the Presumptuous Philosopher also reject the Simulation Argument.My argument is quite long but the concept is not hard to understand.

Different part of the argument can be found here: I, II, III, IV

First thing I want to say is that I do not have a mathematics or philosophy degree. I come from an engineering background. So please forgive me when I inevitably messed up some concept. Another thing I want to mention is that English is not my first language. If you think there is any part that is poorly described please point them out. I would try my best to explain what I meant. That being said, I believe I found a good explanation for the SBP.

My main argument is that in case of the sleeping beauty problem, agents free to communicate thus having identical information can still rightfully have different credence to the same proposition. This disagreement is purely caused by the difference in their perspective. And due to this perspective disagreement, SIA and SSA are both wrong because they are answering the question from an outsider's perspective which provides different answer from beauty's. I concluded that the correct answer should be double-halving.

My argument involves three thought experiments. Here I am breaking it into several parts to facilitate easier discussion. The complete argument can also be found at www.sleepingbeautyproblem.com. However do note it is quite lengthy and not very well written due to my language skills.

First experiment:Duplicating Beauty (DB)

Beauty falls asleep as usual. The experimenter tosses a fair coin before she wakes up. If the coin landed on T then a perfect copy of beauty will be produced. The copy is precise enough that she cannot tell if herself is old or new. If the coin landed on H then no copy will be made . The beauty(ies) will then be randomly put into two identical rooms. At this point another person, let's call him the Selector, randomly chooses one of the two rooms and enters. Suppose he saw a beauty in the chosen room. What should the credence for H be for the two of them?

For the Selector this is easy to calculate. Because he is twice more likely to see a beauty in the room if T, simple bayesian updating gives us his probability for H as 1/3.

For Beauty, her room has the same chance of being chosen (1/2) regardless if the coin landed on H or T. Therefore seeing the Selector gives her no new information about the coin toss. So her answer should be the same as in the original SBP. If she is a halfer 1/2, if she is a thirder 1/3.

This means the two of them would give different answers according to halfers and would give the same answer according to thirders. Notice here the Selector and Beauty can freely communicate however they want, they have the same information regarding the coin toss. So halving would give rise to a perspective disagreement even when both parties share the same information.

This perspective disagreement is something unusual (and against Aumann's Agreement Theorem), so it could be used as an evidence against halving thus supporting Thirdrism and SIA. I would show the problems of SIA in the second thought experiment. For now I want to argue that this disagreement has a logical reason.

Let's take a frequentist's approach and see what happens if the experiment is repeated, say 1000 times. For the Selector, this simply means someone else go through the potential cloning 1000 times and each time let him chooses a random room. On average there would be 500 H and T. He would see a beauty for all 500 times after T and see a beauty 250 times after H. Meaning out of the 750 times 1/3 of which would be H. Therefore he is correct in giving 1/3 as his answer.

For beauty a repetition simply means she goes through the experiment and wake up in a random room awaiting the Selector's choice again.  So by her count, taking part in 1000 repetitions means she would recall 1000 coin tosses after waking up.  In those 1000 coin tosses there should be about 500 of H and T each. She would see the Selector about 500 times with equal numbers after T or H. Therefore her answer of 1/2 is also correct from her perspective.

(Edit Aug 1. I think this is the easiest way to see the relative frequency from beauty's perspective: Ignore the clones first, if someone experienced 1000 coin toss she would expect to see 500 Heads. Now consider how the cloning would affect her conclusion. If there is a T somewhere along the tosses thus a clone is created, both the original and the clone would have remembered exactly the same tosses happened earlier. They would also have the same expectation about coin tosses after they split. So the cloning  does not changes her answer. Therefore after beauty wakes up remembering a large number of tosses she shall expect about half of those landed in Heads. She does not have to consider when she is physically cloned or if she is the original to reach this conclusion.

Alternatively, a more complicated way would be to think about all the coin tosses happened and her own position. For example a beauty is experiencing two coin tosses there are several ways this can happen from a third party's perspective:

1. H-H: Both coin tossing landed on H. There is a 1/4 chance for this happening. Beauty experience 2 Hs.

2. H-T: H and then T. There is a 1/4 chance for this happening. Doesn't matter if the beauty is any one of the two at the end she experienced 1H and 1T.

3. T-(H,H): First toss landed on T and both resulting beauties experience H at the second toss. There is 1/8 chance, doesn't matter if beauty is any one of the two, she experienced 1 H and 1T.

4. T-(H,T): First coin landed on T and a H and T happened at the second round of tosses respectively. There is 1/4 chance. Half the time (1/8) beauty experienced 1H and 1T, while the other half (1/8) she experienced 2 Ts.

5. T-(T,T): All tosses landed on T. There is 1/8 chance, doesn't matter which second toss is hers beauty experienced 2Ts.

The expected number of H from beauty's perspective: (1/4)x2+(1/4)x1+(1/8)x1+(1/8)x1=1: half the number of tosses. Obviously this calculation can be generalized to a greater number of tosses.

Both the above methods correctly gives the relative frequency from beauty's perspective. The most common mistake is for a beauty to think she is equally likely to be any one of the beauties at the end of multiple coin tosses. This seemingly intuitive conclusion is valid from a third party's (the selector's) perspective, aka if he randomly chooses one from all the ending beauties then all beauties are in symmetrical position. However from beauty's first person perspective the ending beauties are not in symmetrical position especially since they have experienced different coin toss results. edit finish)

If we call the creation of a new beauty a "branch off", here we see that from Selector's perspective experiments from all branches are considered a repetition. Where as from Beauty's perspective only experiment from her own branch is counted as a repetition. This difference leads to the disagreement.

This disagreement can also be demonstrated by betting odds. In case of T, choosing any of the two rooms leads to the same observation for the Selector: he always sees a beauty and enters another bet. However, for the two beauties the Selector's choice leads to different observations: whether or not she can see him and enters another bet. So the Selector is twice more likely to enter a bet than any Beauty in case of T, giving them different betting odds respectively.

(EDIT Aug 1. I am aware there are lots of discussions about bets with beauty. Eg. When is the bet settled, base on whose record is it settled, when does beauty pays for the bets and when can she cancels the bet etc. If we consider the importance of perspective disagreement and both party must agree on the bet then these is only one simple way to setup the bet. The bet is settled after each coin toss, it is payed and received at the same time. Beauty's money clones with her. This way cumulatively both selector's and beauties's money would be in accordance with their win/loss record. If this bet is entered whenever the selector sees a beauty in the chosen room then the different betting odds mentioned above would observed. edit ends)

The above reasoning can be easily applied to original SBP. Conceptually it is just an experiment where its duration is divided into two parts by a memory wipe in case of T. The exact duration of the experiment, whether it is two days or a week or five hours, is irrelevant. Therefore from beauty’s perspective to repeat the experiment means her subsequent awakenings need to be shorter to fit into her current awakening. For example, if in the first experiment the two possible awakenings happen on different days, then the in the next repetition the two possible awakening can happen on morning and afternoon of the current day. Further repetitions will keep dividing the available time. Theoretically it can be repeated indefinitely in the form of a supertask. By her count half of those repetitions would be H. Comparing this with an outsider who never experiences a memory wipe: all repetitions from those two days are equally valid repetitions. The disagreement pattern remains the same as in the DB case.

PS: Due to the length of it I'm breaking this thing into several parts. The next part can be found here. Part III here.

## Doomsday argument for Anthropic Decision Theory

3 09 September 2015 12:37PM

tl;dr: there is no real Doomsday argument in ADT. Average utilitarians over-discount the future compared with total utilitarians, but ADT can either increase or decrease this effect. The SIA Doomsaday argument can also be constructed, but this is simply a consequence of total utilitarian preferences, not of increased probability of doom.

I've been having a lot of trouble formulating a proper version of the doomsday argument for Anthropic Decision Theory (ADT). ADT mimics SIA-like decisions (for total utilitarians, those with a population independent utility function, and certain types of selfish agents), and SSA-like decisions (for average utilitarians, and a different type of selfish agent). So all paradoxes of SIA and SSA should be formulatable in it. And that is indeed the case for the presumptuous philosopher and the Adam and Eve paradox. But I haven't found a good formulation of the Doomsday argument.

And I think I know why now. It's because the Doomsday argument-like effects come from the preferences of those average utilitarian agents. Adding anthropic effects does not make the Doomsday argument stronger! It's a non-anthropic effect of those preferences. ADT may allow certain selfish agents to make acausal contracts that make them behave like average utilitarian agents, but it doesn't add any additional effect.

## Doomsday decisions

Since ADT is based on decisions, rather than probabilities, we need to formulate the Doomsday argument in decision form. The most obvious method is a decision that affects the chances of survival of future generations.

But those decisions are dominated by whether the agent desires future generations or not! Future generations of high average happiness are desired, those of lower average happiness are undesirable. This effect dominates the decisions of average utilitarians, making it hard to formulate a decision that addresses 'risk of doom' in isolation. There is one way of doing this, though: looking at how agents discount the future.

## Discounting the future

Consider the following simple model. If humanity survives for n generations, there will have been a total of Gqn humans who ever lived, for some G (obviously q>1). At each generation, there is an independent probability p of extinction, and pq < 1 (so the expected population is finite). At each generation, there is an (independent) choice of consuming a resource to get X utilities, or investing it for the next generation, who will automatically consume it for rX utilities.

Assume we are now at generation n. From the total utilitarian perspective, consuming the resource gives X with certainty, and rX with probability p. So the total utilitarian will delay consumption iff pr>1.

The average utilitarian must divide by total population. Let C be the current expected reciprocal of the population. Current consumption gives an expected XC utilities. By symmetry arguments, we can see that, if humanity survives to the next generation (an event of probability p), the expected reciprocal of population is C/q. If humanity doesn't survive, there is no delayed consumption; so the expected utility of delaying consumption is prXC/q. Therefore the average utilitarian will delay consumption iff pr/q > 1.

So the average utilitarian acts as if they discounted the future by p/q, while the total utilitarian discounts it by p. In a sense, the average utilitarian seems to fear the future more.

But where's the ADT in this? I've derived this result just by considering what an average utilitarian would do for any given n. Ah, but that's because of the particular choice I've made for population growth and risk rate. A proper ADT average utilitarian would compute the different pi and qi for all generation steps and consider the overall value of "consume now" decisions. In general, this could result in discounting that is either higher or lower than the myoptic, one-generation only, average utilitarian. The easy way to see this is to imagine that p is as above (and p is small), as are almost all the q - except for qn. Then the ADT average utilitarian discount rate is still roughly p/q, while the myoptic average utilitarian discount rate at generation n is p/qn, which could be anything.

So the "Doomsday argument" effect - the higher discounting of the future - is an artefact of average utilitarianism, while the anthropic effects of ADT can either increase or decrease this effect.

## SIA Doomsday

LessWronger turchin reminded me of Katja Grace's SIA doomsday argument. To simplify this greatly, it's the argument that since SIA prefers worlds with many people in them (most especially many people "like us"), this increases the probability that there are/were/will be many civilizations at our level of development. Hence the Great Filter - the process that stops the universe from being filled with life - is most likely in the future for our kind of civilizations. Hence the probability of doom is higher.

How does this work, translated into ADT format? Well, imagine there were two options: either the great filter is in the distant evolutionary past, or is in the future. The objective uncertainty is 50-50 on either possibility. If the great filter is in the future, your civilization has a probability p of getting through it (thus there is a total probability of p/2 of your civilization succumbing to a future great filter). You have the option of paying a cost C to avoid the great filter entirely for your civilization. You derive a benefit B from your civilization surviving.

Then you will pay C iff C<Bp/2. But now imagine that you are a total utilitarian, you also care about the costs and benefits from other civilizations, and you consider your decision is linked with theirs via ADT. If the great filter is early, let's assume that your civilization is the only one still in existence. If the great filter is late, then there are Ω civilizations still around.

Therefore if the great filter is early, the total cost is C (your civilization, the only one around, pays C, but gets no benefit as there is no late great filter). However, if the great filter is late, the total cost is ΩC and the total benefit is ΩBp (all of Ω civilizations pay C and get benefit B with probability p). So the expected utility gain is ΩBp-(Ω+1)C. So you will pay the cost iff C < BpΩ/(Ω+1).

To an outsider this looks like you believe the probability of a late great filter is Ω/(Ω+1), rather than 0.5. However, this is simply a consequence of your total utilitarian preferences, and don't reflect an objectively larger chance of death.

## Of all the SIA-doomsdays in the all the worlds...

4 18 October 2013 12:56PM

Ideas developed with Paul Almond, who kept on flogging a dead horse until it started showing signs of life again.

## Doomsday, SSA and SIA

Imagine there's a giant box filled with people, and clearly labelled (inside and out) "(year of some people's lord) 2013". There's another giant box somewhere else in space-time, labelled "2014". You happen to be currently in the 2013 box.

Then the self-sampling assumption (SSA) produces the doomsday argument. It works approximately like this: SSA has a preference for universe with smaller numbers of observers (since it's more likely that you're one-in-a-hundred than one-in-a-billion). Therefore we expect that the number of observers in 2014 is smaller than we would otherwise "objectively" believe: the likelihood of doomsday is higher than we thought.

What about the self-indication assumption (SIA) - that makes the doomsday argument go away, right? Not at all! SIA has no effect on the number of observers expected in the 2014, but increases the expected number of observers in 2013. Thus we still expect that the number of observers in 2014 to be lower than we otherwise thought. There's an SIA doomsday too!

## Enter causality

What's going on? SIA was supposed to defeat the doomsday argument! What happens is that I've implicitly cheated - by naming the boxes "2013" and "2014", I've heavily implied that these "boxes" figuratively correspond two subsequent years. But then I've treated them as independent for SIA, like two literal distinct boxes.

## UFAI cannot be the Great Filter

37 22 December 2012 11:26AM

[Summary: The fact we do not observe (and have not been wiped out by) an UFAI suggests the main component of the 'great filter' cannot be civilizations like ours being wiped out by UFAI. Gentle introduction (assuming no knowledge) and links to much better discussion below.]

### Introduction

The Great Filter is the idea that although there is lots of matter, we observe no "expanding, lasting life", like space-faring intelligences. So there is some filter through which almost all matter gets stuck before becoming expanding, lasting life. One question for those interested in the future of humankind is whether we have already 'passed' the bulk of the filter, or does it still lie ahead? For example, is it very unlikely matter will be able to form self-replicating units, but once it clears that hurdle becoming intelligent and going across the stars is highly likely; or is it getting to a humankind level of development is not that unlikely, but very few of those civilizations progress to expanding across the stars. If the latter, that motivates a concern for working out what the forthcoming filter(s) are, and trying to get past them.

One concern is that advancing technology gives the possibility of civilizations wiping themselves out, and it is this that is the main component of the Great Filter - one we are going to be approaching soon. There are several candidates for which technology will be an existential threat (nanotechnology/'Grey goo', nuclear holocaust, runaway climate change), but one that looms large is Artificial intelligence (AI), and trying to understand and mitigate the existential threat from AI is the main role of the Singularity Institute, and I guess Luke, Eliezer (and lots of folks on LW) consider AI the main existential threat.

The concern with AI is something like this:

1. AI will soon greatly surpass us in intelligence in all domains.
2. If this happens, AI will rapidly supplant humans as the dominant force on planet earth.
3. Almost all AIs, even ones we create with the intent to be benevolent, will probably be unfriendly to human flourishing.

Or, as summarized by Luke:

... AI leads to intelligence explosion, and, because we don’t know how to give an AI benevolent goals, by default an intelligence explosion will optimize the world for accidentally disastrous ends. A controlled intelligence explosion, on the other hand, could optimize the world for good. (More on this option in the next post.)

So, the aim of the game needs to be trying to work out how to control the future intelligence explosion so the vastly smarter-than-human AIs are 'friendly' (FAI) and make the world better for us, rather than unfriendly AIs (UFAI) which end up optimizing the world for something that sucks.

### 'Where is everybody?'

So, topic. I read this post by Robin Hanson which had a really good parenthetical remark (emphasis mine):

Yes, it is possible that the extremely difficultly was life’s origin, or some early step, so that, other than here on Earth, all life in the universe is stuck before this early extremely hard step. But even if you find this the most likely outcome, surely given our ignorance you must also place a non-trivial probability on other possibilities. You must see a great filter as lying between initial planets and expanding civilizations, and wonder how far along that filter we are. In particular, you must estimate a substantial chance of “disaster”, i.e., something destroying our ability or inclination to make a visible use of the vast resources we see. (And this disaster can’t be an unfriendly super-AI, because that should be visible.)

This made me realize an UFAI should also be counted as an 'expanding lasting life', and should be deemed unlikely by the Great Filter.

Another way of looking at it: if the Great Filter still lies ahead of us, and a major component of this forthcoming filter is the threat from UFAI, we should expect to see the UFAIs of other civilizations spreading across the universe (or not see anything at all, because they would wipe us out to optimize for their unfriendly ends). That we do not observe it disconfirms this conjunction.

[Edit/Elaboration: It also gives a stronger argument - as the UFAI is the 'expanding life' we do not see, the beliefs, 'the Great Filter lies ahead' and 'UFAI is a major existential risk' lie opposed to one another: the higher your credence in the filter being ahead, the lower your credence should be in UFAI being a major existential risk (as the many civilizations like ours that go on to get caught in the filter do not produce expanding UFAIs, so expanding UFAI cannot be the main x-risk); conversely, if you are confident that UFAI is the main existential risk, then you should think the bulk of the filter is behind us (as we don't see any UFAIs, there cannot be many civilizations like ours in the first place, as we are quite likely to realize an expanding UFAI).]

A much more in-depth article and comments (both highly recommended) was made by Katja Grace a couple of years ago. I can't seem to find a similar discussion on here (feel free to downvote and link in the comments if I missed it), which surprises me: I'm not bright enough to figure out the anthropics, and obviously one may hold AI to be a big deal for other-than-Great-Filter reasons (maybe a given planet has a 1 in a googol chance of getting to intelligent life, but intelligent life 'merely' has a 1 in 10 chance of successfully navigating an intelligence explosion), but this would seem to be substantial evidence driving down the proportion of x-risk we should attribute to AI.

What do you guys think?

## Why (anthropic) probability isn't enough

19 13 December 2012 04:09PM

A technical report of the Future of Humanity Institute (authored by me), on why anthropic probability isn't enough to reach decisions in anthropic situations. You also have to choose your decision theory, and take into account your altruism towards your copies. And these components can co-vary while leaving your ultimate decision the same - typically, EDT agents using SSA will reach the same decisions as CDT agents using SIA, and altruistic causal agents may decide the same way as selfish evidential agents.

## Anthropics: why probability isn't enough

This paper argues that the current treatment of anthropic and self-locating problems over-emphasises the importance of anthropic probabilities, and ignores other relevant and important factors, such as whether the various copies of the agents in question consider that they are acting in a linked fashion and whether they are mutually altruistic towards each other. These issues, generally irrelevant for non-anthropic problems, come to the forefront in anthropic situations and are at least as important as the anthropic probabilities: indeed they can erase the difference between different theories of anthropic probability, or increase their divergence. These help to reinterpret the decisions, rather than probabilities, as the fundamental objects of interest in anthropic problems.

## SIA, conditional probability and Jaan Tallinn's simulation tree

11 12 November 2012 05:24PM

If you're going to use anthropic probability, use the self indication assumption (SIA) - it's by far the most sensible way of doing things.

Now, I am of the strong belief that probabilities in anthropic problems (such as the Sleeping Beauty problem) are not meaningful - only your decisions matter. And you can have different probability theories but still always reach the decisions if you have different theories as to who bears the responsibility of the actions of your copies, or how much you value them - see anthropic decision theory (ADT).

But that's a minority position - most people still use anthropic probabilities, so it's worth taking a more through look at what SIA does and doesn't tell you about population sizes and conditional probability.

This post will aim to clarify some issues with SIA, especially concerning Jaan Tallinn's simulation-tree model which he presented in exquisite story format at the recent singularity summit. I'll be assuming basic familiarity with SIA, and will run away screaming from any questions concerning infinity. SIA fears infinity (in a shameless self plug, I'll mention that anthropic decision theory runs into far less problems with infinities; for instance a bounded utility function is a sufficient - but not necessary - condition to ensure that ADT give you sensible answers even with infinitely many copies).

But onwards and upwards with SIA! To not-quite-infinity and below!

## SIA does not (directly) predict large populations

One error people often make with SIA is to assume that it predicts a large population. It doesn't - at least not directly. What SIA predicts is that there will be a large number of agents that are subjectively indistinguishable from you. You can call these subjectively indistinguishable agents the "minimal reference class" - it is a great advantage of SIA that it will continue to make sense for any reference class you choose (as long as it contains the minimal reference class).

The SIA's impact on the total population is indirect: if the size of the total population is correlated with that of the minimal reference class, SIA will predict a large population. A correlation is not implausible: for instance, if there are a lot of humans around, then the probability that one of them is you is much larger. If there are a lot of intelligent life forms around, then the chance that humans exist is higher, and so on.

In most cases, we don't run into problems with assuming that SIA predicts large populations. But we have to bear in mind that the effect is indirect, and the effect can and does break down in many cases. For instance imagine that you knew you had evolved on some planet, but for some odd reason, didn't know whether your planet had a ring system or not. You have managed to figure out that the evolution of life on planets with ring systems is independent of the evolution of life on planets without. Since you don't know which situation you're in, SIA instructs you to increase the probability of life on ringed and on non-ringed planets (so far, so good - SIA is predicting generally larger populations).

And then one day you look up at the sky and see:

## SIA fears (expected) infinity

6 12 November 2012 05:23PM

It's well known that the Self-Indication Assumption (SIA) has problems with infinite populations (one of the reasons I strongly recommend not using the probability as the fundamental object of interest, but instead the decision, as in anthropic decision theory).

SIA also has problems with arbitrarily large finite populations, at least in some cases. What cases are these? Imagine that we had these (non-anthropic) probabilities for various populations:

p0, p1, p2, p3, p4...

Now let us apply the anthropic correction from SIA; before renormalising, we have these weights for different population levels:

0, p1, 2p2, 3p3, 4p4...

To renormalise, we need to divide by the sum 0 + p1 + 2p2 + 3p3 + 4p4... This is actually the expected population! (note: we are using the population as a proxy for the size of the reference class of agents who are subjectively indistinguishable from us; see this post for more details)

So using SIA is possible if and only if the (non-anthropic) expected population is finite (and non-zero).

Note that it is possible for the anthropic expected population to be infinite! For instance if pj is C/j3, for some constant C, then the non-anthropic expected population is finite (being the infinite sum of C/j2). However once we have done the SIA correction, we can see that the SIA-corrected expected population is infinite (being the infinite sum of some constant times 1/j).

## The REAL SIA doomsday

8 07 September 2012 11:11AM

Many thanks to Paul Almond for developing the initial form of this argument.

My previous post was somewhat confusing and potentially misleading (and the idea hadn't fully gelled in my mind). But here is a much easier way of seeing what the SIA doomsday really is.

Imagine if your parents had rolled a dice to decide how many children to have. Knowing only this, SIA implies that the dice was more likely to have been a "6" that a "1" (because there is a higher chance of you existing in that case). But, now following the family tradition, you decide to roll a dice for your children. SIA now has no impact: the dice is equally likely to be any number. So SIA predicts high numbers in the past, and no preferences for the future.

This can be generalised into an SIA "doomsday":

• Everything else being equal, SIA implies that the population growth rate in your past is likely to be higher than the rate in the future; i.e. it predicts an observed decline, not in population, but in population growth rates.

## SIA doomsday

4 06 September 2012 04:01PM

Edit: the argument is presented more clearly in a subsequent post.

Many thanks to Paul Almond for developing the initial form of this argument.

It is well known in these circles that the self-sampling assumption (SSA) leads to the doomsday argument. The self-indication assumption (SIA) was developed to counter the doomsday argument. This is a old debate; but what is interesting is that SIA has its own doomsday argument - of a rather interesting and different form.

To see this, let's model the population of a planet somewhat like Earth. From century to century, the planet's population can increase, decrease or stay the same with equal probability. If it increase, it will increase by one billion two thirds of the time, and by two billion one third of the time - and the same for decreases (if it overshoots zero, it stops at zero). Hence, each year, the probability of population change is:

 Pop level change Probability +2 Billion +1 Billion +0 Billion -1 Billion -2 Billion 1/9 2/9 3/9 2/9 1/9

During the century of the Three Lice, there were 3 billion people on the planet. Two centuries later, during the century of the Anchovy, there will still be 3 billion people on the planet. If you were alive on this planet during the intermediate century (the century of the Fruitbat), and knew those two facts, what would your estimate be for the current population?

From the outside, this is easy. The most likely answer if that there is still 3 billion in the intermediate century, which happens with probability 9/19 (= (3/9)*(3/9), renormalised). But there can also be 4 or 2 billion, with probabilities 4/19 each, or 5 or 1 billion, with probabilities 1/19 each. The expected population is 3 billion, as expected.

Now let's hit this with SIA. This weighs the populations by their sizes, changing the probabilities to 5/57, 16/57, 27/57, 8/57and 1/57, for populations of five, four, three, two and one billion respectively. Larger populations are hence more likely; the expected population is about 3.28 billion.

(For those of you curious about what SSA says, that depends on the reference class. For the reference class of people alive during the century of the Fruitbat, it gives the same answer as the outside answer. As the reference class increases, it moves closer to SIA.)

## SIA doomsday

So SIA tells us that we should expect a spike during the current century - and hence a likely decline into the next century. The exact numbers are not important: if we know the population before our current time and the population after, then SIA implies that the current population should be above the trendline. Hence (it seems) that SIA predicts a decline from our current population (or a least a decline from the current trendline) - a doomsday argument.

Those who enjoy anthropic reasoning can take a moment to see what is misleading about that statement. Go on, do it.

Go on.

## [LINK] How Hard is Artificial Intelligence? The Evolutionary Argument and Observation Selection Effects

14 29 August 2011 05:27AM

If you're interested in evolution, anthropics, and AI timelines -- or in what the Singularity Institute has been producing lately -- you might want to check out this new paper, by SingInst research fellow Carl Shulman and FHI professor Nick Bostrom.

The paper:

How Hard is Artificial Intelligence? The Evolutionary Argument and Observation Selection Effects

The abstract:

Several authors have made the argument that because blind evolutionary processes produced human intelligence on Earth, it should be feasible for clever human engineers to create human-level artificial intelligence in the not-too-distant future.  This evolutionary argument, however, has ignored the observation selection effect that guarantees that observers will see intelligent life having arisen on their planet no matter how hard it is for intelligent life to evolve on any given Earth-like planet.  We explore how the evolutionary argument might be salvaged from this objection, using a variety of considerations from observation selection theory and analysis of specific timing features and instances of convergent evolution in the terrestrial evolutionary record.  We find that a probabilistic version of the evolutionary argument emerges largely intact once appropriate corrections have been made.

I'd be interested to hear LW-ers' takes on the content; Carl, too, would much appreciate feedback.