For instance, the principle implies that for any population consisting of very good lives there is a better population consisting of just one person leading a life at a slightly higher level of well-being (Parfit 1984 chapter 19).

I agree with this conclusion in principle, but I need to point out some caveats. Although the argument says a world with one person would have better average quality of life, it is implied that the world would be worse due to loneliness. A world with one person would have to make up for this in some other way. More importantly, going from our current world with lots of people to a world with one person would require killing lots of people, which is unacceptable.

More dramatically, the principle also implies that for a population consisting of just one person leading a life at a very negative level of well-being, e.g., a life of constant torture, there is another population which is better even though it contains millions of lives at just a slightly less negative level of well-being (Parfit 1984).

I hadn't considered this before. The only decent rebuttal I can think of is to claim that negative utility lives (ones not worth living) are fundamentally different from positive utility lives (ones worth living). My first impulse is to maximize average positive utility but minimize total negative utility. Unfortunately, making this distinction raises questions about math between the two types of lives. I would say that minimizing negative utility lives trumps maximizing average positive utility, but I'm pretty sure that would make it hard to choose TORTURE instead of DUST SPECKS.