I think social distance is key here. But perhaps it has more to do with feedback mechanisms supplying information to those wielding power. A village chieftain would have more direct channels of feedback than a general in the Pentagon, say.

Institutionalized power relationships would also create stimuli that didn't exist in the past. For example, prisoners wouldn't remain prisoners for years on end in hunter gatherer societies. They'd either be killed, exiled, or absorbed into the community.

The number of people seems a very likely factor. That could be involved with feedback. It's probably a lot harder to process feedback information from 50,000 people than for 50.

EDIT: A metric might combine the number of people over which one has power, with an economic measure of social distance and social feedback. It's the confluence of these which seems to be the problem, not just the number of persons ruled.