I typically argue against consequentialism based on appeals to intuition and its implications, which are only "consequences" in the sense used by consequentialism if you do some fancy equivocating.
The reason you can't torture people is that those members of your population who aren't as dumb as bricks will realize that the same could happen to them. Such anxiety among the more intelligent members of your society should outweigh the fun experienced by the more easily amused.
Pfft. It is trivially easy to come up with thought experiments where this isn't the case. You can increase the ratio of bricks-to-brights until doing the arithmetic leads to the result that you should go ahead and torture folks. You can choose folks to torture on the basis of well-publicized, uncommon criteria, so that the vast majority of people rightly expect it won't happen to them or anyone they care about. You can outright lie to the population, and say that the people you torture are all volunteers (possibly even masochists who are secretly enjoying themselves) contributing to the entertainment of society for altruistic reasons. Heck, after you've tortured them for a while, you can probably get them to deliver speeches about how thrilled they are to be making this sacrifice for the common morale, on the promise that you'll kill them quicker if they make it convincing.
All that having been said, there are consequentialist theories that do not oblige or permit the torture of some people to amuse the others. Among them are things like side-constraints rights-based consequentialisms, certain judicious applications of deferred-hedon/dolor consequentialisms, and negative utilitarianism (depending on how the entertainment of the larger population cashes out in the math).
"What's the worst that can happen?" goes the optimistic saying. It's probably a bad question to ask anyone with a creative imagination. Let's consider the problem on an individual level: it's not really the worst that can happen, but would nonetheless be fairly bad, if you were horribly tortured for a number of years. This is one of the worse things that can realistically happen to one person in today's world.
What's the least bad, bad thing that can happen? Well, suppose a dust speck floated into your eye and irritated it just a little, for a fraction of a second, barely enough to make you notice before you blink and wipe away the dust speck.
For our next ingredient, we need a large number. Let's use 3^^^3, written in Knuth's up-arrow notation:
3^^^3 is an exponential tower of 3s which is 7,625,597,484,987 layers tall. You start with 1; raise 3 to the power of 1 to get 3; raise 3 to the power of 3 to get 27; raise 3 to the power of 27 to get 7625597484987; raise 3 to the power of 7625597484987 to get a number much larger than the number of atoms in the universe, but which could still be written down in base 10, on 100 square kilometers of paper; then raise 3 to that power; and continue until you've exponentiated 7625597484987 times. That's 3^^^3. It's the smallest simple inconceivably huge number I know.
Now here's the moral dilemma. If neither event is going to happen to you personally, but you still had to choose one or the other:
Would you prefer that one person be horribly tortured for fifty years without hope or rest, or that 3^^^3 people get dust specks in their eyes?
I think the answer is obvious. How about you?