I think you're on the right track. I believe that a small population with high utility per capita is better than a large one with low utility per capita, even if the total utility is larger in the small population. But I think tying that moral intuition to the average utility of the population might be the wrong way to go about it, if only because it creates problems like the one CarlShuman mentioned.

I think a better route might be to somehow attach a negative number to the addition of more people after a certain point, or something like that. Or you can add a caveat that basically says for the system to act like total utilitarianism while the average is negative, and average when it's positive.

Btw, in your original post you mention that we'd need a caveat to stop people from killing existing people to raise the average. A simple solution to that would be to continue to count people in the average even after they are dead.