the result of a process complex enough that it's very difficult to predict the outcome or identify the root cause.
Difficult's a two-place word, and so I'm not sure it makes much sense to argue about whether or not something is 'objectively' difficult, instead of difficult at various states of knowledge.
it carries with it an assumption that disowning gay sons has evolutionary roots, or is ingrained behavior in humans, or is common.
It's not quite that disowning gay sons has evolutionary roots, but that disowning gay sons is not so heavily disfavored as to be extincted. For example, cultures where childbirth is prohibited mostly die out, and so on. But even less obvious things that have an effect on reproductive success are strongly motivating; in cultures with prohibitions against masturbation, those prohibitions are mostly not followed; in cultures where doctors tell mothers to avoid touching their infants because of disease risk, those prohibitions are mostly not followed, and so on. (The impulse for mothers to touch their babies seems very strong, and also very healthy--it actually lowers disease risk by informing the mother what antibodies she needs to produce for her child, and seems critical for proper psychological development.)
And traditional behavior gives us an imperfect window into the economics of the past, which is what's under discussion when we talk about historical selective fitness. If gay sons were helpful enough with nephews and nieces that it was as if they had had their own children, it seems to me they would be welcomed and lauded as examples of loving selflessness. But if gay sons were reproductively disadvantageous, and in particular if it was reasonable to expect that homosexuality is contagious, then there's little cost and some reproductive benefit to forcing them out of the home.
(I should note that the hypothesis that one gene causes both female fecundity and male homosexuality is also consistent with disowning gay sons, but I think that one has other challenges.)
After all, my family didn't disown me, and it seems like disowning gay sons is becoming increasingly uncommon.
Thankfully, people are much more motivated today by individual and relationship satisfaction, neither of which disowning is helpful with. (My family didn't disown me either.)
Does it make sense for families to disown children for being the wrong religion?
It suspect it made sense for religions to disown members that fail to disown their children for being the wrong religion.
Does it make sense for families to disown children for being pregnant?
It looks like control over sexuality was a big deal, and as a first-order effect it seems that signals of that control would heavily impact someone's price on the sexual marketplace. As a second-order effect, it seems that the harsher penalties are for not being controlled, the more likely people are to submit to control. But in less status-stratified societies? Probably not--and it seems like this is mostly a class thing in the societies that I'm familiar with.
This is mostly my speculation, though--I haven't read much on evolutionary accounts of how parents should respond to teenage pregnancy in various environments. I expect someone has thought about this problem.
And traditional behavior gives us an imperfect window into the economics of the past, which is what's under discussion when we talk about historical selective fitness.
I think we should keep in mind just how far back we're talking. I'm not saying we inherited homosexuality from our common ancestor with the modern fruit fly, but at least our common ancestor with other great apes. Framing the question as why would it be selected for in the context of human societies is probably wrong, when what we want to know is why it wasn't sufficiently selected against...
Epistemic status: speculating about things I'm not familiar with; hoping to be educated in the comments. This post is a question, not an answer.
ETA: this comment thread seems to be leading towards the best answer so far.
There's a question I've seen many times, most recently in Scott Alexander's recent links thread. This latest variant goes like this:
Obligate male homosexuality greatly harms reproductive fitness. And so, the argument goes, there must be some other selection pressure, one great enough to overcome the drastic effect of not having any children. The comments on that post list several other proposed answers, all of them suggesting a tradeoff vs. a benefit elsewhere: for instance, that it pays to have some proportion of gay men who invest their resources in their nieces and nephews instead of their own children.
But how do we know if this is a valid question - if the situation really needs to be explained at all?
For obvious political and social reasons, it's hard to be sure how many people are homosexual. Note that we are interested only in obligate homosexuality - bisexuals presumably don't have strongly reduced fitness. The Wikipedia article doesn't really distinguish obligate homosexuality from bi-, pan- and even trans-sexuals. The discussion in the SSC comments used an (unsourced?) range of 1%-3%, which seems at least consistent with other sources, so let's run with that.
The rate of major birth defects in the US, as reported by the CDC, is also about 3%. This counts both developmental and genetic problems, and includes everything from anencephaly (invariably fatal) through Down syndrome (severe but survivable) to cleft palates (minor). But most of these, at least 1.5% of births, were always fatal before modern medicine, and many of the others reduced fitness (via mate selection, if nothing else). Various other defects and diseases, which only manifest later in life, are also thought to be influenced or determined during early development. And so is sexual preference.
(Whether homosexuality is a developmental disorder is not the point; I'm comparing the effect of selection pressure on fatal teratology with its effect on reduced-fitness homosexuality.)
Embryological development is a complex and fragile process, and there are many ways for it to go wrong. We don't wonder how it is possible that selection pressure allows anencephaly to occur in 1 in 4859 births. There are certainly direct causes of anencephaly, explanations of why it happens when it does, but (I think) we don't a priori expect them to be due to tradeoffs yielding benefits elsewhere. It's just as plausible that the tradeoffs involved are against even worse (counterfactual) problems elsewhere - or that there are just no available mutations that don't have these or equally severe problems.
Could it be that linking sexual preference to the biological gender is, for some complex developmental reason, fragile enough that it goes wrong despite all selection pressure to the contrary, that it has no redeeming qualities from the viewpoint of evolution, and that is all there is to it?
When faced with any phenotype with reduced fitness, how can we judge if there is something to be explained - a beneficial tradeoff elsewhere to search for - or merely a hard problem evolution couldn't solve completely? And is there a way to quantify this question, relating it to the known mathematical models of genetics?
Notes:
1. I'm posting this in the spirit of recent suggestions to post more and accept lower quality of (our own) posts to Discussion.
2. I'm going to sleep now and will start replying to comments about 10 hours from now; sorry for the inconvenience.