I think fetishes and homosexuality (which are probably the same phenomenon) arise when these assumptions break down and you place your concepts in unexpected places. For example, the concept of "man" and "woman" are symmetrical enough that it may be possible to switch their locations, depending upon your experiences as a young child.

Several things point towards there being some kind of in utero differentiation correlated with sexuality (fraternal birth order effect, twin studies, etc.) and also gendered behavior / gender identity (twin studies, hormone exposure in primates, studies where children got raised as the opposite gender.) So it seems like whether you'll be gay is something more hardwired, or boostrapped from hardwired parts.