I think your analysis of the 'gladiatorial objection' misses something:

I hope I'm not caricaturing you too much if I condense your rebuttal as follows: "People wouldn't really enjoy watching gladiators fight to the death. In fact, they'd be sickened and outraged. Therefore, utilitarianism does not endorse gladiatorial games after all."

But there's a problem here: If the negative reaction to gladiatorial games is itself partly due to analyzing those games in utilitarian terms then we have a feedback loop.

Games are outrageous --> decrease utility --> are outrageous --> etc.

But this could just as well be a 'virtuous circle':

Games are noble --> increase utility --> are noble --> etc.

If we started off with a society like that of ancient Rome, couldn't it be that the existence of gladiatorial games is just as 'stable' (with respect to the utilitarian calculus) as their non-existence in our own society?

Couldn't it be that we came to regard such bloodsports as being 'immoral' for independent, non-utilitarian reasons*? And then once this new moral zeitgeist became prevalent, utilitarians could come along and say "Aha! Far from being 'fun', just look at how much outrage the games would generate. If only our predecessors had been utilitarians, we could have had avoided all this ugly carnage."

(Perhaps you will bite the bullet here, and grant that there could be a society where gladiatorial games are 'good' by utilitarian standards. But then there doesn't seem to be much hope for a utilitarian justification of the idea that, insofar as we have outlawed bloodsports, we have 'progressed' to a better state of affairs.

Or perhaps you will say that bloodsports would always be judged 'bad' under ideal rational reflection (that is, they go against our CEV). I think this is a much stronger reply, but it's not clear that CEV actually makes sense (i.e. that the limit is well-defined).)

* Sadly my knowledge of history is too meagre to venture an account of how this actually happened.

I have many other objections to utilitarianism up my sleeve. To give the gist of a few of them:

1. Utilitarian calculations are impossible in practice because the future cannot be predicted sufficiently far.
2. Utilitarian calculations are impossible even if theory because outcomes are incommensurable. The indeterminacies concerning whether 'more people' are preferable to 'happier people', and how far a superbeing's happiness is 'worth more' than a human's, are special cases of this, but incommensurability is ubiquitous. (For instance, just try weighing up all of the effects of the decision to buy a car rather than use public transport. The idea is there is a Right Answer Out There seems to me an article of blind faith.)
3. Utilitarianism holds 'terminal preferences' to be beyond reproach. It does not allow for the possibility that an entire self-contained society ought to change its system of preferences, no matter how 'brutal' and 'destructive' these preferences are. (The point about gladiatorial games is a special case of this). It denies that one can make an objective judgement as to whether a paperclipper is 'wrong' and/or 'stupid' to fill the universe with paperclips. Ultimately, might makes right in the struggle between humanity and clippy.
4. Utilitarianism faces some awkward choices in how it values the lives of 'ordinary people' (people who live reasonably happy lives but do not make lasting 'achievements' e.g. progress in science). If their value is positive then apparently it would be better to fill the universe with them than not, which seems absurd. How is it worthwhile or noble to try to explore the entire 'soap opera of Babel'? Isn't it just a stationary stochastic process? Haven't you seen it all once you've seen the first few billion episodes? But if their value is zero (resp. negative) then it seems that nuking an entire planet full of 'ordinary people', assuming it's not the only such planet, is morally neutral (resp. desirable). The only way of resolving the contradiction of human life being both incredibly precious and utterly worthless is to deny that the premise that we need to assign it some value in order to decide how to act.