However, people who talk about race are never talking about genetic clusters, as a rule, they're talking about skin color

Not quite. People usually call Colin Powell an African-American even though his skin isn't that dark.