If you believe the DA, and you also believe you're being simulated (with some probability), then you should believe to be among the last N% humans in the simulation. So you don't escape the DA entirely.
However, it may be that if you believe yourself to be likely in a simulation, you shouldn't believe the DA at all. The DA assumes you know how many humans lived before you, and that you're not special among them. Both may be false in a simulation of human history: it may not have simulated all the humans and pre-humans who ever lived, and/or you may be in a ...
If I were doing it, I'd save computing power by only simulating the people who would program the AI. I don't think I'm going to do that, so it doesn't apply to me. Eliezer doesn't accept the Doomsday Argument, or at least uses a decision theory that makes it irrelevant, so it wouldn't apply to him.
So - I am still having issues parsing this, and I am persisting because I want to understand the argument, at least. I may or may not agree, but understanding it seems a reasonable goal.
The builders know, of course, that this is much riskier than it seems, because its success would render their own observations extremely rare.
The success of the self-modifying AI would make the builders of that AI's observations extremely rare... why? Because the AI's observations count, and it is presumably many orders of magnitude faster?
For a moment, I will assume I...
See LW wiki's Doomsday Argument for reference.
The problem I have with this kind of reasoning is that it causes early reasoners to come to wrong conclusions (though 'on average' the reasoning is most probably true).
Nope. I don't think ignoring causality to such extent makes sense. Simulating many instances of humanity won't make other risks magically go away, because it basically has no effect on them.
Yet another example of how one can misuse rationality and start to believe bogus statements.
Seems backwards. If you are a society that has actually designed and implemented an AI and infrastructure capable of "creating billions of simulated humanities" - it seems de-facto you are the "real" set, as you can see the simulated ones, and a recursive nesting of such things should, in theory have artifacts of some sort (ie. a "fork bomb" in the unix parlance).
I rather think that pragmatically, if a simulated society developed an AI capable of simulating society in sufficient fidelity, it would self-limit - either the simul...
I think you're wrong about "backwards probability".
Probabilities describe your state of knowledge (or someone else's, or some hypothetical idealized observer's, etc.). It is perfectly true that "your" probability for some past event known to you will be 1 (or rather something very close to 1 but allowing for the various errors you might be making), but that isn't because there's something wrong with probabilities of past events.
Now, it often happens that you need to consider probabilities that ignore bits of knowledge you now have. Here's a simple example.
I have a 6-sided die. I am going to roll the die, flip a number of coins equal to the number that comes up, and tell you how many heads I get. Let's say the number is 2. Now I ask you: how likely is it that I rolled each possible number on the die?
To answer that question (beyond the trivial observation that clearly I didn't roll a 1) one part of the calculation you need to do is: how likely was it, given a particular die roll but not the further information you've gained since then, that I would get 2 heads? You will get completely wrong answers if you answer all those questions with "the probability is 1 because I know it was 2 heads".
(Here's how the actual calculation goes. If the result of the die roll was k, then Pr(exactly 2 heads) was (k choose 2) / 2^k, which for k=1..6 goes 0, 1/4, 3/8, 6/16, 10/32, 15/64; since all six die rolls were equiprobable to start with, your odds after learning how many heads are proportional to these or (taking a common denominator) to 0 : 16 : 24 : 24 : 20 : 15, so e.g. Pr(roll was 6 | two heads) is 15/99 = 5/33. Assuming I didn't make any mistakes in the calculations, anyway.)
The SSA-based calculations work in a similar way.
I am not claiming that you should agree with SSA. But the mere fact that it employs these backward-looking probabilities is not an argument against it; if you disagree, you should either explain why computations using "backward probabilities" correctly solve the die+coins problem (feel free to run a simulation to verify the odds I gave) despite the invalidity of "backward probabilities", or else explain why the b.p.'s used in the doomsday argument are fundamentally different from the ones used in the die+coins problem.
A self-modifying AI is built to serve humanity. The builders know, of course, that this is much riskier than it seems, because its success would render their own observations extremely rare. To solve the problem, they direct the AI to create billions of simulated humanities in the hope that this will serve as a Schelling point to them, and make their own universe almost certainly simulated.
Plausible?