Thanks Michael. On another note, I've recommended some of your essays to others, so thanks for writing them as well. 

Matt Leifer, who works in quantum foundations, espouses a view that's probably more extreme than Eric Raymond's above to argue why the effectiveness of math in the natural sciences isn't just reasonable but expected-by-construction. In his 2015 FQXi essay Mathematics is Physics Matt argued that 

... mathematics is a natural science—just like physics, chemistry, or biology—and that this can explain the alleged “unreasonable” effectiveness of mathematics in the physical sciences. 

The main challenge for this view is to explain how mathematical theories can become increasingly abstract and develop their own internal structure, whilst still maintaining an appropriate empirical tether that can explain their later use in physics. In order to address this, I offer a theory of mathematical theory-building based on the idea that human knowledge has the structure of a scale-free network and that abstract mathematical theories arise from a repeated process of replacing strong analogies with new hubs in this network. 

This allows mathematics to be seen as the study of regularities, within regularities, within . . . , within regularities of the natural world. Since mathematical theories are derived from the natural world, albeit at a much higher level of abstraction than most other scientific theories, it should come as no surprise that they so often show up in physics. 

... mathematical objects do not refer directly to things that exist in the physical universe. As the formalists suggest, mathematical theories are just abstract formal systems, but not all formal systems are mathematics. Instead, mathematical theories are those formal systems that maintain a tether to empirical reality through a process of abstraction and generalization from more empirically grounded theories, aimed at achieving a pragmatically useful representation of regularities that exist in nature. 

(Matt notes as an aside that he's arguing for precisely the opposite of Tegmark's MUH.) 

Why "scale-free network"?

It is common to view the structure of human knowledge as hierarchical... The various attempts to reduce all of mathematics to logic or arithmetic reflect a desire view mathematical knowledge as hanging hierarchically from a common foundation. However, the fact that mathematics now has multiple competing foundations, in terms of logic, set theory or category theory, indicates that something is wrong with this view. 

Instead of a hierarchy, we are going to attempt to characterize the structure of human knowledge in terms of a network consisting of nodes with links between them... Roughly speaking, the nodes are
supposed to represent different fields of study. This could be done at various levels of detail. ... Next, a link should be drawn between two nodes if there is a strong connection between the things they represent. Again, I do not want to be too precise about what this connection should be, but examples would include an idea being part of a wider theory, that one thing can be derived from the other, or that there exists a strong direct analogy between the two nodes. Essentially, if it has occurred to a human being that the two things are strongly related, e.g. if it has been thought interesting enough to do something like publish an academic paper on the connection, and the connection has not yet been explained in terms of some intermediary theory, then there should be a link between the corresponding nodes in the network.

If we imagine drawing this network for all of human knowledge then it is plausible that it would have
the structure of a scale-free network. Without going into technical details, scale-free networks have a small number of hubs, which are nodes that are linked to a much larger number of nodes than the average. This is a bit like the 1% of billionaires who are much richer than the rest of the human population. If the knowledge network is scale-free then this would explain why it seems so plausible that knowledge is hierarchical. In a university degree one typically learns a great deal about one of the hubs, e.g. the hub representing fundamental physics, and a little about some of the more specialized subjects that hang from it. As we get ever more specialized, we typically move away from our starting hub towards more obscure nodes, which are nonetheless still much closer to the starting hub than to any other hub. The local part of the network that we know about looks much like a hierarchy, and so it is not surprising that physicists end up thinking that everything boils down to physics whereas sociologists end up thinking that everything is a social construct. In reality, neither of these views is right because the global structure of the network is not a hierarchy.

As a naturalist, I should provide empirical evidence that human knowledge is indeed structured as a scale-free network. The best evidence that I can offer is that the structure of pages and links on the Word Wide Web and the network of citations to academic papers are both scale free [13]. These are, at best, approximations of the true knowledge network. ... However, I think that these examples provide evidence that the information structures generated by a social network of finite beings are typically scale-free networks, and the knowledge network is an example of such a structure.

As an aside, Matt's theory of theory-building explains (so he claims) what mathematical intuition is about: "intuition for efficient knowledge structure, rather than intuition about an abstract mathematical world". 

So what? How does this view pay rent?

Firstly, in network language, the concept of a “theory of everything” corresponds to a network with one enormous hub, from which all other human knowledge hangs via links that mean “can be derived from”. This represents a hierarchical view of knowledge, which seems unlikely to be true if the structure of human knowledge is generated by a social process. It is not impossible for a scale-free network to have a hierarchical structure like a branching tree, but it seems unlikely that the process of knowledge growth would lead uniquely to such a structure. It seems more likely that we will always have several competing large hubs and that some aspects of human experience, such as consciousness and why we experience a unique present moment of time, will be forever outside the scope of physics. 

Nonetheless, my theory suggests that the project of finding higher level connections that encompass more of human knowledge is still a fruitful one. It prevents our network from having an unwieldy number of direct links, allows us to share more common vocabulary between fields, and allows an individual to understand more of the world with fewer theories. Thus, the search for a theory of everything is not fruitless; I just do not expect it to ever terminate.

Secondly, my theory predicts that the mathematical representation of fundamental physical theories will continue to become increasingly abstract. The more phenomena we try to encompass in our fundamental theories, the further the resulting hubs will be from the nodes representing our direct sensory experience. Thus, we should not expect future theories of physics to become less mathematical, as they are generated by the same process of generalization and abstraction as mathematics itself. 

Matt further develops the argument that the structure of human knowledge being networked-not-hierarchical implies that the idea that there is a most fundamental discipline, or level of reality, is mistaken in Against Fundamentalism, another FQXi essay published in 2018. 

Thanks, good example.

Thanks! Added to the list.

(To be honest, to first approximation my guess mirrors yours.) 

Scott Alexander's Mistakes, Dan Luu's Major errors on this blog (and their corrections), Gwern's My Mistakes (last updated 11 years ago), and Nintil's Mistakes (h/t @Rasool) are the only online writers I know of who maintain a dedicated, centralized page solely for cataloging their errors, which I admire. Probably not coincidentally they're also among the thinkers I respect the most for repeatedly empirically grounding their reasoning. Some orgs do this too, like 80K's Our mistakes, CEA's Mistakes we've made, and GiveWell's Our mistakes. 

While I prefer dedicated centralized pages like those to one-off writeups for long content benefit reasons, one-off definitely beats none (myself included). In that regard I appreciate essays like Holden Karnofsky's Some Key Ways in Which I've Changed My Mind Over the Last Several Years (2016), Denise Melchin's My mistakes on the path to impact (2020), Zach Groff's Things I've Changed My Mind on This Year (2017), and this 2013 LW repository for "major, life-altering mistakes that you or others have made", as well as by orgs like HLI's Learning from our mistakes.

In this vein I'm also sad to see mistakes pages get removed, e.g. ACE used to have a Mistakes page (archived link) but now no longer do.

Can you say more about what you mean? Your comment reminded me of Thomas Griffiths' paper Understanding Human Intelligence through Human Limitations, but you may have meant something else entirely. 

Griffiths argued that the aspects we associate with human intelligence – rapid learning from small data, the ability to break down problems into parts, and the capacity for cumulative cultural evolution – arose from the 3 fundamental limitations all humans share: limited time, limited computation, and limited communication. (The constraints imposed by these characteristics cascade: limited time magnifies the effect of limited computation, and limited communication makes it harder to draw upon more computation.) In particular, limited computation leads to problem decomposition, hence modular solutions; relieving the computation constraint enables solutions that can be objectively better along some axis while also being incomprehensible to humans.

I'm mainly wondering how Open Phil, and really anyone who uses fraction of economically-valuable cognitive labor automated / automatable (e.g. the respondents to that 2018 survey; some folks on the forum) as a useful proxy for thinking about takeoff, tracks this proxy as a way to empirically ground their takeoff-related reasoning. If you're one of them, I'm curious if you'd answer your own question in the affirmative?

Thanks for the pointer to that paper, the abstract makes me think there's a sort of slow-acting self-reinforcing feedback loop between predictive error minimisation via improving modelling and via improving the economy itself.

re: weather, I'm thinking of the chart below showing how little gain we get in MAE vs compute, plus my guess that compute can't keep growing far enough to get MAE < 3 °F a year out (say). I don't know anything about advancements in weather modelling methods though; maybe effective compute (incorporating modelling advancements) may grow indefinitely in terms of the chart.

Visual representation of what you mean (imagine the red border doesn't strictly dominate blue) from an AI Impacts blog post by Katja Grace: