Yeah, I don't see why the size of the group is the crucial factor, particularly since the relationship between concentrated wealth and power and size is nonmonotonic.

The usual explanation, which seems sounder, is that when technology became advanced enough to produce a social surplus of wealth, it became possible for a privileged ruling layer to be supported from production by the rest of the population. The possibility of surplus wealth drove stratification.