When people say that it's "too fast," they are making a quantitative claim. The Flynn effect is a standard deviation per generation. Under your scenario of no selection on women, this would require that the bottom half of the bell curve to have no biological children. 50% cuckoldry, perfectly correlated with IQ? Even men who think they've been cuckolded don't have that high a rate.