But if prejudice isn't perfectly even or measuring the same things then you'd expect it to vary.

Yep. And depending on both the strength of the prejudice and the degree of variation, it would show up more on the "nature" or "environment" side. Which is why I'm putting those words in scare quotes.