From what little I know of human neurology, male sex-stuffs are the basic things in the ancient parts of the brain that most any higher mammal share, while female sex-stuffs are much later and closer to our large frontal lobes. (This might be entirely incorrect, please correct me if I am.)

Based on my (also limited) understanding, I think it's partly correct. The brain circuits that create arousal in females are just about as ancient as in males, I think, but in males those circuits can be activated by pretty basic stimuli (i.e. looking at a human with boobs), whereas females may require a lot more complex, nuanced stuff, like emotional closeness–just looking at a naked, muscular guy won't suffice.