When I point out the low-hanging fruit effect to LWers, I do usually get a lot of agreement (and it is appreciated!) but I am starting to wish that someone would dig up some strong contrary evidence.

When the topic of apparent genius deficits and scientific stagnation comes up, people often present multiple explanations, like

1. intrinsic difficulties in scaling scientific activity
2. failure to identify/recognize contemporary scientific successes
3. no more low-hanging fruit
4. bureaucratization and institutional degradation

but tend to present only anecdotal evidence for each — myself included. And I'm not sure that can be helped; I don't know of readily available evidence which powerfully discriminates between the different explanations.

PhilGoetz has data on scientific & technological progress, but I get the impression that much of it's basically time series of counts of inventions & discoveries, which would establish only the whats and not the whys. Likewise, I think I could substantiate my January comment that cohort explains a substantial part of the variation in scientific eminence. And when I scraped together the data, ran the big regression, and found that birth year accounted for (suppose) 30% of the variance in eminence, that wouldn't refute any of the potential explanations for why cohort correlated with eminence.

A partisan of the scaling hypothesis might say, "Obviously, as science gets bigger over time, it gets less efficient; more recently born scientists just lost the birth year draw".

Someone arguing that scientific stagnation is illusory might say, "Obviously, this is a side effect of overlooking more recent scientific geniuses; scientists are working as effectively as before but we don't recognize that thanks to increasing specialization, or our own complacency, or the difficulty of picking out individual drops from the flood of brilliance, or the fact that we only recognize greatness decades after the fact".

I would say, if I were the kind of person who threw the word "obviously" around willy-nilly, "How many times do you expect general relativity to be invented? Obviously, there are only so many simple but important problems to work on, and when we turn to much harder problems, we make slower and more incremental progress".

Someone most concerned with institutional degradation might say, "Obviously, as science has become more bureaucratic and centralized, that's rendered it more careerist, risk-averse & narrow-minded and less ambitious, so of course later generations of scientists would end up being less eminent, because they're not tackling big scientific questions like they did before".

And we don't get anywhere because each explanation is broadly consistent with the observed facts, and each seems obvious to someone.