Is there a justification for not-squaring?

Personally, I am unconvinced that there is any fundamental justification for considering anyone's utility but one's own. But, if you have reason to respect the principles of democracy, the median stands out as the unique point acceptable to a majority. That is, if you specify any other point, a majority would vote to replace that point by the median.

What's the appropriate metric on the space of preferences?

That depends on what kinds of preferences you are comparing. If you are looking at the preferences of a single person, the standard construction of that person's utility function sets the "metric". But if you attempt to combine the preferences of two people, you either need to use the Nash Bargaining solution or Harsanyi's procedure for interpersonal comparison. The first gives a result that is vaguely median-like. The second gives an answer that is suitable for use with the mean.