So my theory is that much of compartmentalization is simply because the search space is so large that people don't end up seeing that there might be a connection between two domains. Even if they do see the potential, or if it's explicitly pointed out to them, they might still not know enough about the domain in question (such as in the example of heavy boots), or they might find the proposed connection implausible.

If a person's knowledge is highly compartmentalized, and consists of these three facts:

1. A human being walked across the moon.

2. There are small rocks on the surface.

3. The moon is a planetary body.

without any educational background, would they choose the right answer?

I believe that their is a high probability that basic intuition would lead to an accurate answer.

So what went wrong, in your case? I don't think that you can attribute it to a failure of compartmentalization. It wasn't that you didn't make connections to your prior knowledge; the problem was that you made too many and that you hadn't organized your priors into a confidence hierarchy.

Confusion occurs when tenuous connections are made and lead to an over-analysis of the question. You differ from the person in the hypothetical, because you had prior knowledge of the forces involved. Connections are only helpful when they are made from strong foundational knowledge to new applications. When you are making many connections from a condition of uncertainty, to a new problem, your intuition fails. It results in the assignment of a low confidence level, to each of many connections, while ignoring basic observations or truisms.

It seems, you were confident in the areas of physics, most applicable in this situation; enumerating the atmosphere, gravity, and mass as the most influential. You attempted to remember how these forces interacted and recognized that they had dimensions to them, that you had forgotten. The connections caused your intuition to be replaced by a humbleness, and the go to answer was a balanced combination of forces. Thus the pen would float.

It is clear that you came to the problem with much more information than my hypothetical person, armed with three foundational facts. You too had those three facts, if that was all that was in your moon compartment the intuition would have been clearer.

(I apologize for the presumptions I made in referring to your thought process. This is a situation, in which, we find ourselves frequently.)

I think this post overstates the case a bit. My general impression is that the scientific method "wins" even in economics and that later works are better than earlier works.

Now it might be true that the average macro-economist of today understands less than Keynes did but I'd be hard pressed to say that the best don't understand more. Moreover, there are really great distillers. In macro for example, Hicks distilled Keynes into something that I would consider more useful that the original.

Nonetheless, I think it is correct that someone should be reading the originals. If not there is the propensity for a particular distiller to miss an important insight and then for everyone else to go one missing it.

What this says to me is that there should be rewards to re-discovery. Suppose that I read Adam Smith and rediscover something great. I should be rewarded for that just as much as if I had come up with the idea myself. Afterall, it has the same effect on the current state of knowledge. However, that will not happen.

Rediscovering is not as prestigious as discovering, because it is not as difficult and does not signal intellectual greatness.

progress means replacing one pile of sludge with another fashionable sludge-pile of similar quality.

The methods available to test these various hypotheses seem to have more of an impact on their prominence, than any objective measure of truth. Classical mechanics conformed to observations and could be confirmed by various tests. This led to widespread adoption until, observations were be made that did not fit the theories. Often the theories are available and cover various possible outcomes, all justified by the intuition offered by the current, yet untestable, theories.

This is where the social sciences run into difficulty. Predictions made by the social sciences are confirmed or disproved by the available methods of verification, at the time the predictions are made. These methods of verification evolve at a slower rate than the theories, and are always limited by the dynamic nature of human actors in large groups. Even if we could determine the utility function for everyone in the world, by the time that utility function had been applied and used to test various SS theories, they would have already changed.

It is unlikely that the LHC will produce results, not yet predicted by various physicists. When it does produce results, some thoeries will be proved and some will be disproved. The confirmation of the correct theory, however, is more valuable than 100 potentially correct yet untestable theories.

Mathematics has evolved quickly, for the same reasons that language has evolved, it is testable in its immediate ability to express and be understood. It has a very clean and objective measurement of success.