Your original post does not talk about heritability. So my answer was: heritability. But then I noticed that it was in the post title and I was confused and did not change my answer.
I posted a calculation on SSC. Let’s say that there is a mutation that is spontaneously created in 1 in 10,000, that it has a 10% chance of producing a homosexual phenotype and that the phenotype has 0.9 fitness, that is, yields an average of 1.8 children. Then the fitness of the gene in 0.99. So in equilibrium, the gene only reproduces itself 99%, so the remaining 1% must be made up by the spontaneous mutation. That is, the prevalence is 100x the spontaneous mutation rate. The prevalence is 1% of the population, of which 0.99% inherit the gene and 0.01% spontaneously acquire it. That’s a genotypic prevalence of 1%. The phenotypic prevalence is 0.1%.
I think 1 in 10,000 is the standard rule of thumb mutation rate. For example, achondroplasia (dwarfism) has a spontaneous mutation rate of 3 in 100,000 and an inherited rate of 1 in 100,000. Apparently dwarfs have about 1/4 of replacement fertility. (The fatality of homozygous achondroplasia complicates the situation. The gene definitely has a fitness of 1/4, but if dwarfs only marry dwarfs, the fitness of the people would be 3/8, I think.)
Also, the approach to equilibrium is exponential with base the fitness.
I think 1 in 10,000 is the standard rule of thumb mutation rate.
Its more like 10^-8.
Epistemic status: speculating about things I'm not familiar with; hoping to be educated in the comments. This post is a question, not an answer.
ETA: this comment thread seems to be leading towards the best answer so far.
There's a question I've seen many times, most recently in Scott Alexander's recent links thread. This latest variant goes like this:
Obligate male homosexuality greatly harms reproductive fitness. And so, the argument goes, there must be some other selection pressure, one great enough to overcome the drastic effect of not having any children. The comments on that post list several other proposed answers, all of them suggesting a tradeoff vs. a benefit elsewhere: for instance, that it pays to have some proportion of gay men who invest their resources in their nieces and nephews instead of their own children.
But how do we know if this is a valid question - if the situation really needs to be explained at all?
For obvious political and social reasons, it's hard to be sure how many people are homosexual. Note that we are interested only in obligate homosexuality - bisexuals presumably don't have strongly reduced fitness. The Wikipedia article doesn't really distinguish obligate homosexuality from bi-, pan- and even trans-sexuals. The discussion in the SSC comments used an (unsourced?) range of 1%-3%, which seems at least consistent with other sources, so let's run with that.
The rate of major birth defects in the US, as reported by the CDC, is also about 3%. This counts both developmental and genetic problems, and includes everything from anencephaly (invariably fatal) through Down syndrome (severe but survivable) to cleft palates (minor). But most of these, at least 1.5% of births, were always fatal before modern medicine, and many of the others reduced fitness (via mate selection, if nothing else). Various other defects and diseases, which only manifest later in life, are also thought to be influenced or determined during early development. And so is sexual preference.
(Whether homosexuality is a developmental disorder is not the point; I'm comparing the effect of selection pressure on fatal teratology with its effect on reduced-fitness homosexuality.)
Embryological development is a complex and fragile process, and there are many ways for it to go wrong. We don't wonder how it is possible that selection pressure allows anencephaly to occur in 1 in 4859 births. There are certainly direct causes of anencephaly, explanations of why it happens when it does, but (I think) we don't a priori expect them to be due to tradeoffs yielding benefits elsewhere. It's just as plausible that the tradeoffs involved are against even worse (counterfactual) problems elsewhere - or that there are just no available mutations that don't have these or equally severe problems.
Could it be that linking sexual preference to the biological gender is, for some complex developmental reason, fragile enough that it goes wrong despite all selection pressure to the contrary, that it has no redeeming qualities from the viewpoint of evolution, and that is all there is to it?
When faced with any phenotype with reduced fitness, how can we judge if there is something to be explained - a beneficial tradeoff elsewhere to search for - or merely a hard problem evolution couldn't solve completely? And is there a way to quantify this question, relating it to the known mathematical models of genetics?
Notes:
1. I'm posting this in the spirit of recent suggestions to post more and accept lower quality of (our own) posts to Discussion.
2. I'm going to sleep now and will start replying to comments about 10 hours from now; sorry for the inconvenience.