Epistemic status: I might be reinventing the wheel here
A common cause for rejection of total utilitarianism is that it implies the so-called Repugnant Conclusion, of which a lot has been written elsewhere. I will argue that while this implication is solid in theory, it does not apply in our current known universe. My view is similar to the one expressed here, but I try to give more details.
The Repugnant Conclusion IRL
The greatest relevance of the RC in practice arises in situations of scarce resources and Malthusian population traps¹: We compare population A, where there are few people with each one having plentiful resources, and population Z, which has grown from A until the average person lives in near-subsistence conditions.
Let's formalize this a bit: suppose each person requires 1 unit of resources for living, so that the utility of a person living on 1 resources is exactly 0: a completely neutral life. Furthermore, suppose utility is linear w.r.t. resources: doubling resources means doubling utility and 10 resources correspond to 1 utility. If there are 100 resources in the world, population A might contain 10 people with 10 resources each and total utility 10; population Z might contain 99 people with 100/99 resources each and total utility also 10.
So in this model, we are indifferent between A and Z even as everyone in Z is barely subsisting, and this would be the Repugnant Conclusion². But this conclusion depends crucially on the relationship between resources and utility which we have assumed to be linear. What if our assumption is wrong? What is this relationship in the actual world? Note that this is an empirical question³.
It is well known that self-reported happiness varies logarithmically with income⁴, both between countries and for individuals within each country, so it seems reasonable to assume that the utility-resources relation is logarithmic: exponential increases in resources bring linear increases in utility.
Back to our model, assuming log utility, how do we now compare A and Z? If utility per person is where are the resources available to that person, then total utility is . Assuming equality in the population (see the Equality section), if are total resources and is population size, each person has resources and so we have
We can plot total utility (vertical axis) as a function of N (horizontal axis) for

Here we can see two extremes of cero utility: at where there are no persons and at where each person lives with 1 resources, at subsistence level. In the middle there is a sweet spot, and the maximum M lies at around 37 people⁵.
Now we can answer our question! Population A, where is better than population Z where , but M is a superior alternative to both.
So I have shown that there is a population M greater and better than A where everyone is worse off, how is that different from the RC? Well, the difference is that this does not happen for every population, but only for those where average well being is relatively high. Furthermore, the average individual in M is far above subsistence.
Equality
In my model I assumed an equal distribution of resources over the population, mainly to simplify the calculations, but also because under the log relationship and if the population is held constant, total utilitarianism endorses equality. I will try to give an intuition for this and then a formal proof.

This graph represents individual utility (vertical axis) vs individual resources (horizontal axis). If there are two people, A and B, each having 2.5 and 7.5 resources respectively, we can reallocate resources so that both now are at point M, with 5 each. Note that the increase in utility for A is 3, while the decrease for B is a bit less than 2, so total utility increases by more than 1.
This happens no matter where in the graph are A and B due to the properties of the log function. As long as there is a difference in wealth you can increase total utility by redistributing resources equally.
For a formal proof, see ⁶.
Implications
The main conclusion I get from this is that although total utilitarianism is far from perfect, it might give good results in practice. The Repugnant Conclusion is not dead, however. We can certainly imagine some sentient aliens, AIs or animals whose utility function is such that greater, worse-average-utility populations end up being better. But in this case, should we really call it repugnant? Could our intuition be fine-tuned for thinking about humans, and thus not applicable to those hypothetical beings?
I don't know to what extent have others explored the connection between total utilitarianism and equality, but I was surprised when I realized that the former could imply the latter. Of course, even if total utility is all that matters, it might not be possible to reshuffle it among individuals with complete liberty, which is the case in my model.
Footnotes
1: One might consider other ways of controlling individual utility in a population besides resources (e.g. mind design, torture...) but these seem less relevant to me.
2: Actually, in the original formulation Z is shown to be better than A, not just equally good.
3: As long as utility is well defined, that is. Here I will use self-reported happiness as a proxy for utility.
4: See the charts here
5: We can find the exact maximum for any R with a bit of calculus:
A nice property of this is that the ratio that maximizes is constant for al (the exact constant obtained here is just due to the arbitrary choice of base 10 for the logarithms)
6: For a population of individuals the distribution of resources which maximizes total utility is that where for all . The proof goes by induction on .
This is obvious in the case . For the induction step, we can separate a population of into two sets of and 1 individuals respectively so that total utility is . Suppose we allocate resources to the group of , and to the last person. By hypothesis, each of the people must receive resources to maximize their total utility so
Now we have to decide how much should be.
Therefore, for each of the first individuals and for the last one
Firstly, excellent post! A cool idea, well-written, and very thought-provoking. Some thoughts on the robustness of the result:
Suppose that every individual were able to produce at least 1 unit of resources throughout their life. Then total utility is monotonically increasing in the number of people, and you have the repugnant conclusion again. How likely is this supposition? Assuming we have arbitrarily advanced technology, including AI, humans will be pretty irrelevant to the production of resources like food or compute (if we're in simulation). But plausibly humans could still produce "goods" which are valuable to other humans, like friendship. Let's plug this into your model above and see what happens. I'll assume that humans need at least 1/K physical resources to survive, but otherwise their utility is logarithmic in the amount of physical resources + friendship that they get. Also, assume that every person receives as much friendship as they produce. So
, with an upper bound of N = KR. When F >= 1, then the optimal value of N is in fact KR, and so utility per person is
, which can be arbitrarily close to 0 (depending on K). When 0 <= F < 1, then I think you get something like your original result again, but I'm not sure. Empirically, I expect that the best friends can produce F >> 1, i.e. if you had nothing except just enough food/water to keep yourself alive, but also you were the sole focus of their friendship, then you'd consider your life well worth living. Idk about average production, but hopefully that'll improve in the future too. In summary, friendship may make things repugnant :P
Here's another version of the repugnant conclusion and your argument. Suppose that the amount of resources used by each person is roughly fixed per unit time (because, say, we're living in simulation), but that there's a period of infancy and early childhood which uses up resources and isn't morally valuable. Then the resources used up by one person are I + L, where I is the length of their infancy and L is the length of the rest of their life, but the utility gained from their life is a function of L alone. What function of L? Perhaps you think that it's linear in L - for example, If you're a hedonic utilitarian, it's plausible that people will be just as happy later in their life as earlier. (In fact, right now, old people tend to be happiest). If so, you must endorse the anti-repugnant conclusion, where you'd prefer a population with very few very long-lived people, to minimise the fixed cost of infancy. If you're a preference utilitarian, maybe you think that there's diminishing marginal utility to having your preferences satisfied. It then follows that there's an optimal point at which to kill people, which isn't too soon (otherwise you're incurring high fixed costs) and isn't too late (otherwise people's marginal utility diminishes too much) - a conclusion which is analogous to your result.