Go to a casino. Bet $1 on something with a 50% chance of winning. If you win, you have won $1; try again. If you lose, double your bet size (which means that winning will leave you having won $1 total over the sequence of doubled bets) and repeat.
One argument says that in the long run, you will come out a winner, because every bet you make is part of a sequence and at the end of that sequence, you are $1 richer. Another argument says that in the long run, you will only break even, because each bet has a 50% chance of winning and a 50% chance of losing the same amount of money.
Of course, the answer is that you can't increase your bet infinitely, and when you stop increasing your bet, the statistical loss at the point where you stop increasing your bet exactly makes up for the statistical win all the other times you finished the sequence and won $1.
Furthermore, if you could increase your bet infinitely, this problem wouldn't happen, but if you could increase your bet infinitely, the expectation isn't well defined, because you are trying to compute it for a non-converging infinite series.
All this problem is is the same idea applied to probability of death instead of expectation of win. If the madman ever runs out of people, the overall probability depends exactly on what the madman does when he runs out of people (since it's not as well defined as it is for bets). If the madman never runs out of people, the probability involves a non-converging infinite series and so is not well defined.
If this is a metaphor for extinction, then when the madman runs out of people, he keeps rolling the dice on the remaining people until it eventually comes up snake eyes, in which case the chance of extinction is 100%. On the other hand, they can last arbitrarily long given an arbitrarily small probability of extinction.
An earlier version of my analysis (the previous blog post) looked at the case of finite n and found, as you suggest, that the possibility of running out of people to kidnap is an important consideration. You can choose the number of batches n to be so large that it is virtually certain a priori that the madman will eventually murder:
P(eventually murders) = 1 - epsilon for some small epsilon
However, it turns out that conditioning on the fact that you are kidnapped changes the probability dramatically:
P(eventually murders | you are kidnapped) = about 10/...
I'm sure that many of you here have read Quantum Computing Since Democritus. In the chapter on the anthropic principle the author presents the Dice Room scenario as a metaphor for human extinction. The Dice Room scenario is this:
1. You are in a world with a very, very large population (potentially unbounded.)
2. There is a madman who kidnaps 10 people and puts them in a room.
3. The madman rolls two dice. If they come up snake eyes (both ones) then he murders everyone.
4. Otherwise he releases everyone, then goes out and kidnaps 10 times as many people as before, and returns to step 3.
The question is this: if you are one of the people kidnapped at some point, what is your probability of dying? Assume you don't know how many rounds of kidnappings have preceded yours.
As a metaphor for human extinction, think of the population of this world as being all humans who ever have or ever may live, each batch of kidnap victims as a generation of humanity, and rolling snake eyes as an extinction event.
The book gives two arguments, which are both purported to be examples of Bayesian reasoning:
1. The "proximate risk" argument says that your probability of dying is just the prior probability that the madman rolls snake eyes for your batch of kidnap victims -- 1/36.
2. The "proportion murdered" argument says that about 9/10 of all people who ever go into the Dice Room die, so your probability of dying is about 9/10.
Obviously this is a problem. Different decompositions of a problem should give the same answer, as long as they're based on the same information.
I claim that the "proportion murdered" argument is wrong. Here's why. Let pi(t) be the prior probability that you are in batch t of kidnap victims. The proportion murdered argument relies on the property that pi(t) increases exponentially with t: pi(t+1) = 10 * pi(t). If the madman murders at step t, then your probability of being in batch t is
pi(t) / SUM(u: 1 <= u <= t: pi(u))
and, if pi(u+1) = 10 * pi(u) for all u < t, then this does indeed work out to about 9/10. But the values pi(t) must sum to 1; thus they cannot increase indefinitely, and in fact it must be that pi(t) -> 0 as t -> infinity. This is where the "proportion murdered" argument falls apart.
For a more detailed analysis, take a look at
http://bayesium.com/doomsday-and-the-dice-room-murders/
This forum has a lot of very smart people who would be well-qualified to comment on that analysis, and I would appreciate hearing your opinions.