Suppose I on average found category X of dissimilar people, less attractive because they are different, this would not mean I would find every member less attractive than average.

Okay. That definitely makes more sense.

Also historically, people really haven't been that picky, its amazing to what extent we chose those that are available. While barriers between populations that coexist on the same territory do exist, they are not absolute. And all else being equal the smaller group will quickly become basically a hybrid population, while the larger population will still have a bunch of people who match their previous genetic profile. To give an extreme example If you are the last member of your tribe, the only way you get to mate is to find a partner outside your tribe.

While I can certainly understand why minorities would mix with majorities, that doesn't really explain the opposite.

But I suppose the opposite isn't really that true. After all, looking at America, we see that blacks have lots of european ancestry, but we don't really see whites having lots of african ancestry (unless I'm mistaken). My example with Jews explains how Jews took from the majority populations, not the other way around.

Thanks for the clarification.