It's possible I'm more of a loner than you, so I find the idea of hermits less repugnant.

Maybe I was using too strong a word when I said I found it "repugnant."

be careful you're not imagining these hermits not doing anything fun.

I took your advice and tried to imagine the hermits doing things I like doing when I am alone. That was hard at first, since most of the things I like doing alone still require some other personat some point (reading a book requires an author, for instance). But imagining a hermit studying nature, interacting with plants and animal (the animals obviously have to be bugs and other nonsapient, nonsentient animals to preserve the purity of the scenario, but that's fine with me), doing science experiments, etc, that doesn't seem repugnant at all.

But I still prefer, or am indifferent to, one utility monster hermit vs. many normal hermits, especially if the hermits are all clones living in very similar environments.

On the other hand, clones tend to really mess up my intuitions regardless of the hypothetical. I'm pretty sure they should be penalized for lacking diversity, but as for the actual amount ...

I'm not sure how much I value diversity that isn't appreciated. I think I'd prefer a diverse group of hermits to a nondiverse group, but the fact that the hermits never meet and are unable to appreciate each others diversity seems to make it less valuable to me, the same way a painting that's locked in a room where no one will ever see it is less valuable. That may come back to my belief that value usually needs both an objective and subjective component. On the other hand I might value diversity terminally as well, as I said the fact that no one appreciated the hermit's diversity made it less valuable to me, but not valueless.

Maybe you're hyperbolically discounting that future pleasure and it's outweighed by the temporary displeasure caused by agreeing to something abhorrent? ;)

I think that if an FAI scanned ArisKatsaris' brain, extrapolated values from that, and then was instructed to extrapolate what a non-hyperboli- discounting ArisKatsaris would choose, it would answer that ArisKatsaris would not choose to get rewired to receive pleasure from the end of mankind.

Of course, there's no way to test such a hypothesis.

On the other hand, they do seem a bit ... dull? Lacking the sort of explosive variety I picture in the Good Future.

I agree, I think that the reason that sparsely populated scenarios seem repugnant to us isn't because we want to maximize total utility, and they have a lower total utility level. Rather it's because we value things like diversity, friendship, love, and interpersonal entanglements, and we find the idea of a future where these things do not exist to be repugnant.

One argument hardcore total utilitarians use to claim people have inconsistent preferences about population ethics is that when ranking the following populations:

A) Ten billion people with ten thousand utility each, for a total utility of 100 trillion. B) 200 trillion people with one utility each, for a total utility of 200 trillion. C) One utility monster with 50 trillion utility.

People consider A to be better than both B and C. "Aha!" cry the total utilitarians. "So in one scenario utility is too heavily concentrated, and in another it isn't concentrated enough! Intransitive preferences! Status quo bias!"

What the hardcore total utilitarians fail to realize is that the reason people find C repugnant isn't because utility is heavily concentrated, it's that in order to have such high utility when it is the lone being in the universe, the utility monster must place no value at all on diversity, friendship, love, and interpersonal entanglements, and so forth. C isn't repugnant because utility is too concentrated, or because of status quo bias, it's repugnant because the lone inhabitant of C lacks a large portion of the gifts we give to tomorrow.

To test this theory I decided to compare populations A, B, and C again, with the stipulation that the multitude inhabiting of A and B were all hermits who never saw each other, and instead of diverse individuals they were repeated genetic duplicates of the same person. Sure enough I found all three populations repugnant. But I might have found C to be a little less repugnant than A and B.

Couldn't this argument cut the other way? Maybe the only reason we think a small population with an average utility of 100 is worse than a billion people with an average utility of 99 is that we're "kinked" to a world inhabited by billions.

Personally, when I read "The City and the Stars," which takes place on a very sparsely populated future Earth, I agreed with the author that it was a bad thing that the local population was less ambitious and curious than the humans of the past. But I did not think it was a horrible travesty that there were so few people. I assume that for the duration of my reading I empathized with the inhabitants, and hence found their current population levels desirable. I've noticed the same thing when reading other books set in sparsely populated settings. I wish the inhabitants were better off, but don't think there need to be more of them.

A typical argument against "quality" focused population ethics is that they favor much smaller populations with higher qualities of life than we currently have, while an argument against "quantity" focused population ethics is that they favor much larger populations with lower qualities of life than we currently have. Both of these seem counter-intuitive, but which intuition should be kept and which should be rejected? Considering that our moral intuitions developed in small hunter gatherer bands, I wouldn't be surprised if the quality focused population ethics was actually the correct one.

I'm tempted to choose B just because if I choose A someone will try to use the Axiom of Transitivity to "prove" that I value some very large amount of paperclippers more than some small amount of humans. And I don't.

I might also choose B because the paperclipper might destroy various beautiful nonliving parts of the universe. I'm not sure if I really value beautiful rock formations and such, even if there is no one to view them. I tend to agree that something requires both an objective and subjective component to be truly valuable.

On the other hand, maybe the value for beautiful things I will never see is some sort of "between the margins" value, something that I value, but that my values regarding eudaemonic life are lexically prior to. All other things being equal, I'd prefer a universe with even a tiny amount of eudaemonic life (that isn't suffering or anything like that) to a totally lifeless universe chock-full of unobserved beautiful stuff. But maybe a lifeless pretty universe is more valuable to me than a lifeless ugly universe, all other things being equal.

Sentience is one of the basic goods. If the sysop is non-sentient, then whatever computronium is used in the sysop is, WRT sentience, wasted.

Not necessarily. It depends on what the sysop does with all that computing power once it's in charge. Sentience is one of the basic goods, but having whatever sentient creatures exist live excellent lives is another one. If the sysop uses 60% of the computing power to run itself and 40% to run sentient creatures, it could still be a net win if the sysop spends most of its time finding new ways to make those other creature's lives as wonderful as possible.

Look at it another way, the organic matter currently being used to make your clothes, food, home, etc could probably also be used to make more humans out of. But it's probably better to use it to improve your life then to create a bunch of cold, naked, hungry people.

That's true, but shouldn't we also give weight to the billions of people who might die if we screw up and create some sort dangerous AI? Or, in a less exotic scenario, we end up fighting a war with some kind of world-destroying weapon we invent? We've already had some close calls in that department. So far the amount of benefits the accelerating changes have given us outweigh the harms, but we've been really lucky.

Or, more pertinent to the OP, if the lives that would be lost if we create a bunch of AIs that we don't consider morally significant, erase them, and then later realize we were wrong to consider them not morally significant?

You do, but saying fashion is a money pump because you need to keep spending money to stay in fashion is like saying that grocery stores are a money pump because you need to keep spending money to not starve.

That would be a money pump if they preferred style A over style B, and style B over style A. But that isn't what they really prefer. What they prefer is to wear whatever style is in fashion. Style A is preferable to style B if and only if style A is in fashion, and vice versa. Preferring specific styles is an instrumental value, staying in fashion is the terminal value.

Similarly, imagine one day I prefer pizza to tacos, the next day I prefer tacos to pizza, and the day after that I prefer pizza to tacos. My preferences weren't intransitive, because I didn't really prefer those foods at all, what I preferred was eating tasty things, preferring tacos or pizza were instrumental values. During the day I preferred tacos to pizza, I had gotten sick of pizza and no longer found it tasty.

People aren't 'born' in the normal sense - instead they are 'fluctuated' into existence as full-grown adults. Instead of normal 'death', people simply dissolve painlessly after a given amount of time. Nobody is aware that at some point in the future they will 'die', and whenever someone does all currently existing people have their memories instantly modified to remove any trace of them.

This scenario is way, way worse than the real world we live in. It's bad enough that some of my friends and loved ones are dead. I don't want to lose my memories of them too. The social connections people form with others are one of the most important aspects of their lives. If you kill someone and destroy all their connections at the same time you've harmed them far more badly than if you just killed them.

Plus, there's also the practical fact that if you are unaware of when you will "dissolve" it will be impossible for you to plan your life to properly maximize your own utility. What if you had the choice between going to a good movie today, and a great movie next week, and were going to dissolve tomorrow? If you didn't know that you were going to dissolve you'd pick the great movie next week, and would die having had less fun than you otherwise could have had.

I'd prefer option 1 in this scenario, and in any other, because the title of the OP is a misnomer, people can't be replaced. The idea that you are "replacing" someone if you create a new person after they die implies that people are not valuable, they are merely containers for holding what is really valuable (happiness, utility, etc.), and that it does not matter if a container is destroyed as long as you can make a new one to transfer its contents into. I completely disagree with this approach. Utility is valuable because people are valuable, not the other way around. A world with lower utility where less people have died is better than a world of higher utility with more death.