For most purposes, selection should be on the long time scale. If the environment is uniform, long-term fitness should be the same as short-term fitness (though less random). If there are occasional catastrophes and population bottlenecks, then being adapted to them may be more important than being adapted to the usual environment. Even with a uniform environment, there may be a long tail to male fitness which is not easily observed in the short-term. Genghis Khan demonstrates that there is, at least occasionally, a long tail to human male reproductive success. If genetic factors were relevant and the opportunity arose reasonably often, then we should expect those genes to spread, even if they impede normal reproduction.