I'd argue, as our culture defines race, you really encounter a large number of different and distinct ways of classifying groups of people, of which skin color is just one which gets disproportionate attention owing to historic cultural reasons combined with extreme visual salience (black skin is much easier to notice than eye color).

So how do you account for the fact that race as measured by what you consider the "flawed cultural way" correlates as strongly as it does with things like intelligence and criminality?

In other periods of time, other ways of grouping people by race got more attention.

And quite possibly they were dealing with different populations and the groupings they used did in fact correlate with important things.