The choice between an "averagist" and "totalist" model of optimal human welfare is a tough one. The averagist wants to maximize average happiness (or some such measure of welfare); the totalist wants to maximize total happiness. Both lead to unfortunate reductio arguments. Average human welfare can be improved by eliminating everyone who is below average. This process can be repeated successively until we have only one person left, the happiest man in the world. The totalist would proceed by increasing the population to the very edge of carrying capacity, so that everyone was just a small increment above being so unhappy that they would commit suicide.

Neither model seems to match with our intuitions. As Eliezer has frequently warned, anyone who may accidentally or intentionally create a super-intelligent Artificial Intelligence that might take over the world had better beware of it adopting one of these extremes, even if the intention is to program it to be beneficent.

I'm sure philosophers have considered these questions for centuries. I'd be curious to know if there is a principled model for optimal human happiness which does not conflict so violently with our moral instincts. Not to privilege our instincts unjustifiably; it's possible that the AI might be *right* to adopt one of the views above, and we are wrong, with our muddled human thinking. I would feel better if there were a nice, consistent and relatively simple model which did not lead to a seemingly horrific outcome.