It's generally not a matter of skull shape or of environmental / modifiable factors. Soft tissue is the cause of most facial asymmetry. You know, like one cheek being a little plumper than the other -- that's just asymmetrical fat distribution. Or another common one, eyebrow asymmetry -- the hair follicles are just not positioned in the same way on both sides of the face. Skeletal asymmetry, like jaw bones being positioned at different heights, is somehow rarer in my observations. It most definitely affects attractiveness, though people don't seem to be able to put a finger on exactly what is wrong with a slightly asymmetrical face -- and for that matter, neither can the people with asymmetrical faces themselves; it only hits you when you look at a horizontally flipped image of what you're used to see (for others, when they view one in a mirror for the first time, and for oneself, when one views oneself through two (or any even number of) mirror reflections). My guess is that the brain tends to symmetrize images of familiar faces.

(I haven't read any research either, but I tend to gawk a lot at people.)