Comment Permalink

cousin_it7y80

Thanks for the reply! Can you tell more about the failure to publish ADT? I know that from arxiv, but don't know the details.

Dr_Manhattan7y20

Curious if you're at all updating using MIRIs poor publishing record as evidence of a problem based on the Stuart+Wei's story below. (Seems like trying to get through journal review might be a huge cost and do little to advance knowledge). Or you think this was an outlier or the class of things MIRI should be publishing is less subject for the kinds of problems mentioned.

6Stuart_Armstrong7y

Here's the list of journals submitted to, btw: Philosophical Quarterly, Philosophical Review, Mind, Synthese, Erkenntnis, Journal of Philosophy, Harvard Review of Philosophy.

18Stuart_Armstrong7y

Nothing particularly riveting... Submitted it to a large numbers of philosophy journals in sequence, got either rejections or please modify and resubmit, did re-writings, and these were all eventually rejected. A few times it looked like it might get accepted, but then it was borderline rejected. Basically, they felt it was either not important enough, or they were committed to a probability view of anthropics, and didn't like the decision based approach. Or that I was explaining it badly, but the reviewers were not consistent as to what was bad about the explanations.

See in context

11 The Doomsday argument in anthropic decision theory

by Stuart_Armstrong

31st Aug 2017

4 min read

11

EDIT: added a simplified version here.

Crossposted at the intelligent agents forum.

In Anthropic Decision Theory (ADT), behaviours that resemble the Self Sampling Assumption (SSA) derive from average utilitarian preferences (and from certain specific selfish preferences).

However, SSA implies the doomsday argument, and, to date, I hadn't found a good way to express the doomsday argument within ADT.

This post will remedy that hole, by showing how there is a natural doomsday-like behaviour for average utilitarian agents within ADT.

Anthropic behaviour

The comparable phrasings of the two doomsday arguments (probability and decision-based) are:

In the standard doomsday argument, the probability of extinction is increased for an agent that uses SSA probability versus one that doesn't.
In the ADT doomsday argument, an average utilitarian behaves as if it were a total utilitarian with a higher revealed probability of doom.

Thus in both cases, doomsday agent believes/behaves as if it were a non-doomsday agent with a higher probability of doom.

Revealed probability of events

What are these revealed probabilities?

Well, suppose that X and X' are two events that may happen. The agent has a choice between betting on one or the other; if they bet on the first, they get a reward of r if X happens, if they bet on the second, they get a reward of r' if X' happens.

If an agent is an expected utility maximiser and chooses X over X', this implies that rP(X) ≥ r'P(X'), where P(X) and P(X') are the probabilities the agent assigns to X and X'.

Thus, observing the behaviour of the agent allows one to deduce their probability estimation for X and X'.

Revealed anthropic and non-anthropic probabilities

To simplify comparisons, assume that Y is an event that will happen with probability 1; if the agent bets on Y, it will get a reward of 1. The Y's only purpose is to compare with other events.

Then X is an event that will happen with an unknown probability, if bet on, the agent will get a reward of r. In comparison, X_s is an event that will happen with certainty if and only if humanity survives for a certain amount of time. If the agent bets on X_s and it happens, it will then give a reward of r_s.

The agent need to bet on one of Y, X, and X_s. Suppose that the agent is an average utilitarian, and that their actual estimated probability for human survival is p; thus P(X_s)=p. If humanity survives, the total human population will be Ω; if it doesn't, then it will be limited to ω≤Ω.

Then the following table gives the three possible bets and the expected utility the average utilitarian will derive from them. Since the average utilitarian needs to divide their utility by total population, this expected utility will be a function of the probabilities of the different population numbers.

By varying r and r_s, we can establish what probabilities the agent actually gives to each event, by comparing with situation when it bets of Y. If we did that, but assumed that the agent was a total utilitarian rather than an average one, we would get the apparent revealed probabilities given in the third column:

Bet	Utility	App. rev. prob. if tot.
Y	(1-p)/ω + p/Ω	1
X	rP(X)[(1-p)/ω + p/Ω]	P(X)
X_s	r_s(p/Ω)	p'= (p/Ω) / [(1-p)/ω + p/Ω]

Note that if Ω=ω - if the population is fixed, so that the average utilitarian behaves the same as a total utilitarian - then p' simplifies to (p/ω) / (1/ω) = p, the actual probability of survival.

It's also not hard to see that p' strictly decreases as Ω increases, so it will always be less than p if Ω > ω.

Thus if we interpret the actions of an average utilitarian as if they were a total utilitarian, then for reward conditional on human survival - and only for those rewards, not for others like betting on X - their actions will seem to imply that they give a lower probability of human survival than they actually do.

Conclusion

The standard doomsday argument argues that we are more likely to be in the first 50% of the list of all humans that will ever live, rather than in the first 10%, which is still more likely than us being in the first 1%, and so on. The argument is also vulnerable to changes of reference class; it gives different implications if we consider 'the list of all humans', 'the list of all mammals', or 'the list of all people with my name'. The doomsday argument has no effect on probabilities not connected with human survival.

All these effects reproduce in this new framework. Being in the first n% means that the total human population will be at least ω100/n, so the total population Ω grows as n shrinks -- and p', the apparent revealed probability of survival, shrinks as well. Similarly, average utilitarianism gives different answers depending on what reference class is used to define its population. And the apparent revealed probabilities that are not connected with human survival are unchanged from a total utilitarian.

Thus this seems like a very close replication of the doomsday argument in ADT, in terms of behaviour and apparent revealed probabilities. But note that it is not a genuine doomsday argument. It's all due to the quirky nature of average utilitarianism; the agent doesn't really believe that the probability of survival goes down, they just behave in a way that would make us infer that they believed that, if we saw them as being a total utilitarian. So there is no actual increased risk.

Anthropics

Personal Blog

11

Mentioned in

20Rational Feed

1Simplified Anthropic Doomsday

New Comment

58 comments, sorted by

top scoring

Click to highlight new comments since: Today at 4:12 PM

[-]cousin_it7y140

This might be a tactless question, but can you explain why people should be talking about ADT or FDT? Both of them appeared in 2011 or later, and seem equivalent to UDT1 which came out in 2009 and was superseded by UDT1.1 in 2010 (allowing your copies to take different actions in a coordinated way). Why not agree to call the state of the art UDT1.1, until we come up with some new idea worthy of the name UDT1.2?

[-]Stuart_Armstrong7y80

ADT is nothing but a fragment of UDT applied to anthropic problems, I agree. So from the perspective of decision theory, it's nothing interesting.

However, the results of UDTish approaches in anthropics are interesting for those interested in anthropics. And I decided to present ADT to that audience, because UDT in anthropic situations can be phrased in terms of assumptions that are very natural from the anthropic situation, without needing the full justification of UDT (my initial approach was to use ADT to lure people to UDT, but the failure to publish ADT means this idea fell away ^_^ ).

[-]cousin_it7y80

Thanks for the reply! Can you tell more about the failure to publish ADT? I know that from arxiv, but don't know the details.

[-]Stuart_Armstrong7y180

Or that I was explaining it badly, but the reviewers were not consistent as to what was bad about the explanations.

[-]Wei Dai7y140

Thanks for posting this! I wonder how much negative utility academia causes just in terms of this kind of frustrating experience, and how many kids erroneously start down an academic path because they never hear people tell stories like this.

Here's my own horror story with academic publishing. I was an intern at an industry research lab, and came up with a relatively simple improvement to a widely used cryptographic primitive. I spent a month or two writing it up (along with relevant security arguments) as well as I could using academic language and conventions, etc., with the help of a mentor who worked there and who used to be a professor. Submitted to a top crypto conference and weeks later got back a rejection with comments indicating that all of the reviewers completely failed to understand the main idea. The comments were so short that I had no way to tell how to improve the paper and just got the impression that the reviewers weren't interested in the idea and made little effort to try to understand it. My mentor acted totally unsurprised and just said something like, "let's talk about where to submit it next." That's the end of the story because I decide if that's how academia works I wanted to have nothing to do with it when there's, from my perspective, an obviously better way to do things, i.e., writing up the idea informally, posting it to a mailing list and getting immediate useful feedback/discussions from people who actually understand and are interested in the idea.

[-]Stuart_Armstrong7y80

writing up the idea informally, posting it to a mailing list and getting immediate useful feedback/discussions from people who actually understand and are interested in the idea.

Note that I was doing that as well. And many academics similarly do both routes.

[-]Wei Dai7y40

What's your and their motivation to also do the academic publishing route (or motivation to go into or remain in academia which forces them to also do the academic publishing route)? I guess I can understand people at FHI doing it for the prestige associated with being in academia which they can convert into policy influence, but why do, say, academic decision theorists do it? Do they want the prestige as a terminal goal? Is it the easiest way for them to make a living while also doing research? Did they go down the academic path not knowing it would be like this and it was too late when they found out?

Assuming the above exhausts the main reasons, it seems like a good idea for someone who doesn't care much about the prestige and can make money more easily elsewhere to skip academic publishing and use the time to instead do more research, make more money, or just for leisure?

[-]Stuart_Armstrong7y80

If you're a published academic in field X, you're part of the community of published academics in field X, which, in many but not all cases, is the entirety of people doing serious work in field X. "Prestige" mainly translates to "people who know stuff in this area take me seriously".

[-]Wei Dai7y90

When I was involved in crypto there were forums that both published academics and unpublished hobbyists participated in, and took each other seriously. If this isn't true in a field, it makes me doubt that intellectual progress is still the highest priority in that field. If I were a professional philosopher working in anthropic reasoning, I don't see how I can justify not taking a paper about anthropic reasoning seriously unless it passed peer review by anonymous reviewers whose ideas and interests may be very different from my own. How many of those papers can I possibly come across per year, that I'd justifiably need to outsource my judgment about them to unknown peers?

(I think peer review does have a legitimate purpose in measuring people's research productivity. University admins have to count something to determine who to hire and promote, and number of papers that pass peer review is perhaps one of the best measure we have. And it can also help outsiders to know who can be trusted as experts in a field, which is what I was thinking of by "prestige". But there's no reason for people who are already experts in a field to rely on it instead of their own judgments.)

[-]Stuart_Armstrong7y50

If I were a professional philosopher working in anthropic reasoning, I don't see how I can justify not taking a paper about anthropic reasoning seriously

Depends on how many cranks there are in anthropic reasoning (lots) and how many semi-serious people post ideas that have already been addressed or refuted in papers already (in philosophy in general, this is huge; in anthropic reasoning, I'm not sure).

[-]Wei Dai7y00

Lots of places attract cranks and semi-serious people, including the crypto forums I mentioned, LW, and everything-list which was a mailing list I created to discuss anthropic reasoning as one of the main topics, and they're not that hard to deal with. Basically it doesn't take a lot of effort to detect cranks and previously addressed ideas, and everyone can ignore the cranks and the more experienced hobbyists can educate the less experienced hobbyists.

EDIT: For anyone reading this, the discussion continues here.

[-]Stuart_Armstrong7y00

Basically it doesn't take a lot of effort to detect cranks and previously addressed ideas

This is news to me. Encouraging news.

[-][anonymous]7y00

For anyone reading this, the discussion continues here.

[This comment is no longer endorsed by its author]Reply

[-]turchin7y00

If specially dedicated journal for anthropic reasoning exists (or say for AI risks and other x-risks), it will probably improve quality of peer review and research? Or it will be no morу useful than Lesswrong?

[-]Stuart_Armstrong7y40

If I were a professional philosopher working in anthropic reasoning, I don't see how I can justify not taking a paper about anthropic reasoning seriously

But there are no/few philosophers working in "anthropic reasoning" - there are many working in "anthropic probability", to which my paper is an interesting irrelevance. it's essentially asking and answering the wrong question, while claiming that their own question is meaningless (and doing so without quoting some of the probability/decision theory stuff which might back up the "anthropic probabilities don't exist/matter" claim from first principles).

I expected the paper would get published, but I always knew it was a bit of a challenge, because it didn't fit inside the right silos. And the main problem with academia here is that people tend to stay in their silos.

[-]Wei Dai7y20

But there are no/few philosophers working in "anthropic reasoning" - there are many working in "anthropic probability", to which my paper is an interesting irrelevance. it's essentially asking and answering the wrong question, while claiming that their own question is meaningless

Seems like a good explanation of what happened to this paper specifically.

(and doing so without quoting some of the probability/decision theory stuff which might back up the "anthropic probabilities don't exist/matter" claim from first principles)

I guess that would be the thing to try next, if one was intent on pushing this stuff back into academia.

And the main problem with academia here is that people tend to stay in their silos.

By doing that they can better know what the fashionable topics are, what referees want to see in a paper, etc., which help them maximize the chances of getting papers published. This seems to be another downside of the current peer review system as well as the larger publish-or-perish academic culture.

[-]Stuart_Armstrong7y60

Here's the list of journals submitted to, btw: Philosophical Quarterly, Philosophical Review, Mind, Synthese, Erkenntnis, Journal of Philosophy, Harvard Review of Philosophy.

[-]Dr_Manhattan7y20

[-]cousin_it7y10

I have a lot of respect for Stuart and Wei, so this discussion is very interesting to me. That said, my own experiences with academia were a bit more pleasant, and I know many "normal" folks who regularly get their work published in journals. It certainly takes a ton of "academic market research", which is hard. But in the end I feel that it makes the work stronger and more appealing.

[-]Dr_Manhattan7y20

My hypothesis based on this discussion is that it could be a lot harder to publish stuff outside of current academic mainstream. IIRC Hinton had trouble accepting foundational DL papers in the beginning. MIRI type stuff (and possibly Wei's crypto, though not sure about the details) could have been far enough from the mainstream to increase the hardness a lot.

[-]Lukas_Gloor7y50

I believe that you're right about the historicity, but for me at least, any explanations of UDT I came across a couple of years ago seemed too complicated for me to really grasp the implications for anthropics, and ADT (and the appendix of Brian's article here) were the places where things first fell into place for my thinking. I still link to ADT these days as the best short explanation for reasoning about anthropics, though I think there may be better explanations of UDT now (suggestions?). Edit: I of course agree with giving credit to UDT being good practice.

[-][anonymous]7y00

Wei's explanations were enough for me and others to devise multiple formalizations of UDT based on proof search, halting oracles, modal logic, and logical uncertainty. The first such result came out in 2010 and there have been many since then. A lot of that work happened on the decision-theory-workshop mailing list, on LW, at MIRI workshops, and later on agentfoundations. Here's an example problem given by Gary Drescher in 2010, which is miles more advanced than anything discussed in the context of ADT or FDT to this day. You're probably right that popular explanations of UDT are lacking, but that doesn't seem like an argument to popularize it under a different name...

[This comment is no longer endorsed by its author]Reply

[-][anonymous]7y00

I did not intend my comment as such an argument and am in favor of giving credit to UDT.

[This comment is no longer endorsed by its author]Reply

[-]avturchin3y40

ADT cares about my copy which will have the biggest impact on the future utility and thus favors some kind of utility monster - and this is an analogue of Doomsday argument for ADT.

Example: Under ADT I should assume that I can create FAI and save the world – as in that case I will have the biggest impact on the future utility.

[-]turchin7y20

I created a practical example, which demonstrates me correctness of your point of view as I understand it.

Imagine that there is 1000 civilization in the Universe and 999 from them will extinct in their early stage. And one civilization, which will not extinct, could survive only if it spends billions of billions on large prevention project.

Each civilization independently developed DA argument on its early stage and concluded that Doom probability is almost 1. Each civilization has two options in early stage:

1) Start partying, trying to get as much utility as possible before the inevitable catastrophe. 2) Ignore anthropic update and go all in in desperate attempt of the catastrophe prevention.

If we choose option 1, then all other agents similar to our decision process will come to the same conclusion and even a civilization which was able to survive, will not attempt to survive, and as a result, all intelligent life in the universe will die off.

If we choose 2, we will most likely fail anyway, but one of the civilizations will survive.

The choice depends on our utilitarian perspective: If we interested only in our civilization well-being, option 1 will give us higher utility, but if we care about the survival of other civilizations, we should choose 2, even if we believe that probability is against us.

Is this example correct from the point of ADT?

[-]Stuart_Armstrong7y20

This is a good illustration of anthropic reasoning, but it's an illustration of the presumptuous philosopher, not of the DA (though they are symmetric in a sense). Here we have people saying "I expect to fail, but I will do it anyway because I hope others will succeed, and we all make the same decision". Hence it's the total utilitarian (who is the "SIAish" agent) who is acting against what seems to be the objective probabilities.

http://lesswrong.com/lw/8bw/anthropic_decision_theory_vi_applying_adt_to/

[-]turchin7y10

"So there is no actual increased risk" - but we have to behave like it is? For example, invest in defences, or start partying?

Personally, I always use the following to explain such logic to myself: I don't believe that my vote will change a result of a presidential election, but I have to behave as if it will, and go to vote.

[-]Yosarian27y00

I don't believe that my vote will change a result of a presidential election, but I have to behave as if it will, and go to vote.

The way I think of this is something like this:

There is something like a 1 in 10 million chance that my vote will affect the presidential election (and also some chance of my voting affecting other important elections, like Congress, Governor, ect).

Each year, the federal government spends $3.9 trillion dollars. It's influence is probably actually significantly greater then that, since that doesn't include the effect of laws and regulations and such, but let's go with that number for the sake of argument.

If you assume that both parties are generally well-intended and will mostly use most of that money in ways that create positive utility in one way or another, but you think that party A will do 10% more effectively then party B, that's a difference in utility of $390 billion dollars.

So a 1 in 10 million chance of having a 390 billion dollar effect divides into something like an expected utility of $39,000 for something that will take you maybe half an hour. (Plus, since federal elections are only every 2 years, it's actually double that.)

I could be off by an order of magnitude with any of these estimates, maybe you have a 1 in 100 million chance of making a difference, or maybe one party is only 1% better the the other, but it seems from a utilitarian point of view like it's obviously worth doing even so.

The same logic can probably be used for these kind of existential risks as well.

[-]turchin7y10

Yes, but there are situation when the race is not tight, like 40 to 60, and it is very improbable that my vote will work alone, but if we assume that something like ADT works, all people similar to me will behave as if I command them and total utility will be millions time more - as my vote will turn in million votes of people similar to me.

[-]Yosarian27y20

Yeah, that's a fair point.

[-]Stuart_Armstrong7y00

"So there is no actual increased risk" - but we have to behave like it is?

If you're a consistent average utilitarian, yes. Total utilitarian (or non-indexical utility) no. Selfish utilities are more tricky.

[-]turchin7y10

It is possible to create similar to DA situation and check if using DA will give a correct prediction about probabilities in not so extreme situation as human extinction.

The experiment if following. Imagine, that I don't know the length of a year in months, but I could ask just one person about his date of birth. In my case, it is September, 9th month. Assuming that date of birth is random from the full length of the year, I could calculate that expected length of a year is 18 months with 50 per cent probability, which is not far from 12, and is not 0.01 or 10 000 000. And it like I use mediocrity principle without any application of my utilitarian position, but to predict actual value.

[-]Stuart_Armstrong7y00

Yes. When all the priors are fixed, and what you're updating is known to you, then the style of reasoning behind the DA works (see some of Nick Bostrom's examples).

However, in ADT, I reject the very notion that anthropic probabilities make sense. Things like "which of these almost identical agents are actually you" are things that do not make sense in terms of probability. However, despite that, ADT (simplified anthropic CDT) can still let you make good decisions.

[-]Manfred7y00

Since we are in the real world, it is a possibility that there is a copy of me, e.g. as a boltzmann brain, or a copy of the simulation I'm in.

Does your refusal to assign probabilities to these situations infect everyday life? Doesn't betting on a coin flip require conditioning on whether I'm a boltzmann brain, or am in a simulation that replaces coins with potatoes if I flip them? You seem to be giving up on probabilities altogether.

[-]Stuart_Armstrong7y20

Suppose that there is 50% ehance of there being a boltzmann brain copy of you - that's fine, that is a respectable probability. What ADT ignores are questions like "am I the boltzmann brain or the real me on Earth?" The answer to that is "yes. You are both. And you currently control the actions of both. It is not meaningful to ask 'which' one you are."

Give me a preference and a decision, and that I can answer, though. So the answer to "what is the probability of being which one" is "what do you need to know this for?"

[-]turchin7y00

I agree with this: "yes. You are both. And you currently control the actions of both. It is not meaningful to ask 'which' one you are."

But have the following problem: what if the best course of action for me depends on am I Boltzmann brain or real person? It looks like I still have to update according to which group is larger: real me of Boltzmann brain me.

It also looks like we use "all decision computation processes like mine process" as something like what I called before "natural reference class". And in case of DA it is all beings who thinks about DA.

[-]Stuart_Armstrong7y10

I'll deal with the non-selfish case, which is much easier.

In that case, Earth you and Boltzmann brain you have the same objectives. And most of the time, these objectives make "Boltzmann brain you" irrelevant, as their actions have so consequences (one exception could be "ensure everyone has a life that is on average happy, in which case Earth you should try and always be happy, for the sake of the Boltzmann brain yous). So most of the time, you can just ignore Boltzmann brains in ADT.

Yes, that is a natural reference class in ADT (note that it's a reference class of agents-moments making decisions, not of agents in general; it's possible that someone else is in your reference class for one decision, but not for another).

But "all beings who think about DA" is not a natural reference class, as you can see when you start questioning it ("to what extent do they think about DA? Under what name? Does it matter what conclusions they draw?...)

[-]Manfred7y00

That's not quite what I was talking about, but I managed to resolve my question to my own satisfaction anyhow. The problem of conditionalization can be worked around fairly easily.

Suppose that there is 50% ehance of there being a boltzmann brain copy of you

Actually, the probability that you should assign to there being a copy of you is not defined under your system - otherwise you'd be able to conceive of a solution to the sleeping beauty problem - the entire schtick is that Sleeping Beauty is not merely ignorant about whether another copy of her exists, but that it is supposedly a bad question.

Hm, okay, I think this might cause trouble in a different way that I was originally thinking of. Because all sorts of things are possibilities, and it's not obvious to me how ADT is able to treat reasonable anthropic possibilities different from astronomically-unlikely ones, if it throws out any measure of unlikeliness. You might try to resolve this by putting in some "outside perspective" probabilities, e.g. that an outside observer in our universe would see me as normal most of the time and me as a Boltzmann brain less of the time, but this requires making drastic assumptions about what the "outside observer" is actually outside, observing. If I really was a Boltzmann brain in a thermal universe, an outside observer would think I was more likely to be a Boltzmann brain. So postulating an outside perspective is just an awkward way of sneaking in probabilities gained in a different way.

This seems to leave the option of really treating all apparent possibilities similarly. But then the benefit of good actions in the real world gets drowned out by all the noise from all the unlikely possibilities - after all, for every action, one can construct a possibility where it's both good and bad. If there's no way to break ties between possibilities, no ties get broken.

[-]Stuart_Armstrong7y00

Actually, the probability that you should assign to there being a copy of you is not defined under your system - otherwise you'd be able to conceive of a solution to the sleeping beauty problem

Non-anthropic ("outside observer") probabilities are well defined in the sleeping beauty problem - the probability of heads/tails is exactly 1/2 (most of the time, you can think of these as the SSA probabilities over universes - the only difference being in universes where you don't exist at all). You can use a universal prior or whatever you prefer; the "outside observer" doesn't need to observe anything or be present in any way.

I note that you need these initial probabilities in order for SSA or SIA to make any sense at all (pre-updating on your existence), so I have no qualms claiming them for ADT as well.

[-]Manfred7y00

And what if the universe is probably different for the two possible copies of you, as in the case of the boltzmann brain? Presumably you have to take some weighted average of the "non-anthropic probabilities" produced by the two different universes.

Re: note. This use of SSA and SIA can also be wrong. If there is a correct method for assigning subjective probabilities to what S.B. will see when she looks at outside, it should not be an additional thing on top of predicting the world, it should be a natural part of the process by which S.B. predicts the world.

EDIT: Okay, getting a better understanding of what you mean now. So you'd probably just say that the weight on the different universes should be exactly this non-anthropic probability, assigned by some universal prior or however one assigns probability to universes. My problem with this is that when assigning probabilities in a principled, subjective way - i.e. trying to figure out what your information about the world really implies, rather than starting by assuming some model of the world, there is not necessarily an easily-identifiable thing that is the non-anthropic probability of a boltzmann brain copy of me existing, and this needs to be cleared up in a way that isn't just about assuming a model of the world. If anthropic reasoning is, as I said above, not some add-on to the process of assigning probabilities, but a part of it, then it makes less sense to think something like "just assign probabilities, but don't do that last anthropic step."

But I suspect this problem actually can be resolved. Maybe by interpreting the non-anthropic number as something like the probability that the universe is a certain way (i.e. assuming some sort of physicalist prior), conditional on there only being at least one copy of me, and then assuming that this resolves all anthropic problems?

[-]entirelyuseless7y00

That answer might be fine for copies, but not for situations where copies are involved in no way, like the Doomsday Argument. It is nonsense to say that you are both early and late in the series of human beings.

[-]Stuart_Armstrong7y00

Copies are involved in DA. To use anthropics, you have to "update on your position on your reference class" (or some similar construction). At that very moment, just before you update, you can be any person at all - if not, you can't update. You can be anyone equally.

(of course, nobody really "updates" that way, because people first realise who they are, then long after that learn about the DA. But if SSA people are allowed to "update" like that, I'm allowed to look at the hypothetical before such an update)

[-]entirelyuseless7y00

"I'm allowed to look at the hypothetical before such an update"

In the Doomsday argument as I understand it, you are allowed to do that. Nothing about our present knowledge gives any strong suggestion that human race will last millions of years.

[-]entirelyuseless7y00

As I said about using many lists, it is obvious that Doomsday style reasoning will in general be roughly correct. Arguments to the contrary are just wishful thinking.

[-]Stuart_Armstrong7y00

DA style reasoning in non-anthropic situations are fine. I reject the notion that anthropic probabilities are meaningful. The fact that SIA doesn't have DA, and is in most ways a better probability theory than SSA, is enough to indicate (ha!) that something odd is going on.

[-]entirelyuseless7y00

We've had this discussion before. I see no reason to think anthropic probabilities are meaningless, and I see every reason to think DA style reason will generally work in anthropic situations just as well as in other situations.

[-]turchin7y00

SIA has its own DA via Fermi paradox as K.Grace showed. https://meteuphoric.wordpress.com/2010/03/23/sia-doomsday-the-filter-is-ahead/

I also don't see that you actually reject with probabilities, as I still have to behave as if they were true. (However, I understand the similar logic in the voting example: I have to go to vote for my candidate and should reject any updates that my personal vote very unlikely change result of the election.)

Something like this example may help: I don't believe that the world will end soon, but I have to invest more in x-risks prevention after I learned about DA (and given that I average utilitarian). I think some more concrete example will be useful for understanding here.

[-]Stuart_Armstrong7y00

I looked at the SAI DA in my previous post on DA, and I feel I got that one right:

http://lesswrong.com/lw/mqg/doomsday_argument_for_anthropic_decision_theory/

[-]turchin7y00

There is also a form of DA which at first glance is not anthropic, and it is Gott's formula, where the time of existence of a thing in the future is roughly equal previous time of existence if we observe it in a random moment. Gott's equation gives inverse ratio of probability of survival with time.

While Carter's-Leslie DA is used to update known x-risks probability based on anthropic reasoning about the number of observers, Gott's equation connects time and probability without an explicit account of observer's number.

Gott formula also was tested in practice and showed that it works.

Gott formula could be applied to the time of existence of technological civilization, and here the main controversy starts, as it is not clear if it is applicable here, and what should be counted as the beginning of the civilization.

It also not clear if ADT reasoning is applicable Gott equation, and if not, DA is not killed.

[-]entirelyuseless7y00

As I said, there is no reason why all forms of DA cannot be valid at the same time. They are just different methods of estimation and the conclusions are approximate. But most of them would agree that humans will not survive for billions of years, and that conclusion is virtually certain to be true.

[-]turchin7y00

If we agree that the DA is valid, there may be ways to cheat it. For example, if global population reduced to 1 person, he would escape the power of Carter's DA, (but not Gott's one). This 1 person could be superintelligent AI.

Another cheating option is memory reset or a large number of simulations.

The difference is on decision level. If extinction is inevitable, there is no reason to spend time on any prevention efforts, aтв it is better to start partying now trying to get as much utility as possible before a catastrophe.

But if there is a small probability of survival, such behavior will be maladaptive, as it will kill this small probability for another civilization which could survive if it goes all in in survival mode.

So from the decision point of view, in this case, we should ignore almost inevitable extinction probability provided by DA, - and it is the point which Stuart has been demonstrating in his post. But he didn't suggest this example and it was not clear to me until now.

[-]entirelyuseless7y00

According to the argument here, we are at a nearly perfectly average place in the series of habitable planet years, rather than early in the universe as it first appears. If this is the case, it strongly suggests that the human race will go extinct on earth, rather than moving to any other place, ever. I think this is probably what will happen.

And even if it does not, physics suggests even more strongly that the human race will go extinct sooner or later. I am fairly sure this will happen.

The DA just supports all of those things that we already know: there is no reason except wishful thinking to think that humans will not go extinct in a normal way.

[-]turchin7y00

Totally agree. But in my interpretation of ADT, the DA should not stop us from trying to survive (in a comment above Stuart said that it is not DA, but "presumptuous philosopher" paradox) as there is still a small chance.

I also use what I call Meta Doomsday argument. It basically said that there is a logical uncertainty about if DA or any of its version are true, and thus we should give some subjective probability Ps to the DA is true. Let's say it is 0.5.

As DA is also a probabilistic argument, we should multiply Ps on DA's probability shift, and we will still get a large update in the extinction probability as a result.

[-]entirelyuseless7y20

I agree with all this.

[-]entirelyuseless7y00

"The argument is also vulnerable to changes of reference class; it gives different implications if we consider 'the list of all humans', 'the list of all mammals', or 'the list of all people with my name'. "

There is nothing strange about this, just as there is nothing strange about the fact that there is evidence for and against the same thing, and stronger and weaker evidence. If you took 100 lists like that, and you knew exactly how many of each thing there were going to be, you would turn out to be 50% through the lists on average, even though you might be e.g. only 10% through one particular list, and through 92% of another etc.

[-]turchin7y10

There is only one natural solution to the problem of the reference class: use only those who know about Doomsday argument as the members of the reference class.

Unfortunately, acceptance of this reference class means Doom very soon, as such people appeared only after 1983, and in large numbers even in 2000s. It means that the reference class of people who knows about DA will end in 10-20 years from now.

But it may not mean extinction: may be DA will be disproved or will go out of favor.

[-]entirelyuseless7y00

"as such people appeared only after 1983"

This is false. I thought of the argument myself before I ever heard it from anyone else, and I have no doubt at all that many people did the same before me over the centuries.

[-]Stuart_Armstrong7y00

There is only one natural solution to the problem of the reference class: use only those who know about Doomsday argument as the members of the reference class.

Or discard the idea of natural reference classes entirely, as SIA and ADT both do.

Moderation Log