About the xkcd comic: I am not actually convinced that IS how it works.

The actual evidence is that beliefs about group differences tend to be highly accurate and proportional, see http://www.rci.rutgers.edu/~jussim/socialperception.html