This seems like a pretty good reason not to insist that the aggregation is invariant under relabeling of the individuals.

Also, Harsanyi's social aggregation theorem for a finite number of individuals, states that whenever individual preferences and aggregate preferences can all be stated as utility functions, if the aggregation is indifferent whenever each individual is (or alternatively: if the aggregation prefers A to B whenever each individual does, and if this condition is non-vacuous), then the aggregation is a linear combination of the individual utility functions. It looks to me like it should be possible to generalize this to an infinite population, though I haven't checked the details. If this is true, it would be inconsistent with in-variance under relabeling.