Physics seems to have a bunch of useful epistemic techniques which haven’t been made very legible yet.
The two big legible epistemic techniques in technical fields are Mathematical Proofs, and The Scientific Method. Either derive logically X from some widely-accepted axioms, or hypothesize X and then do a bunch of experiments which we’d expect to come out some other way if X were false. It seems pretty obvious that science requires a bunch of pieces besides those in order to actually work in practice, but those are the two which we’ve nailed down most thoroughly.
Then there’s less-legible methods. Things like fermi estimates, gears-level models, informal mathematical arguments, an aesthetic sense for kinds-of-models-which-tend-to-generalize-well, the habit of figuring out qualitative features of an answer before calculating it, back-of-the-envelope approximations, etc.
Take informal mathematical arguments, for example. We’re talking about things like the use of infinitesimals in early calculus, or delta functions, or Fourier methods, or renormalization. Physicists used each of these for decades or even centuries before their methods were rigorously proven correct. In each case, one could construct pathological examples in which the tool broke down, yet physicists in practice had a good sense for what kinds-of-things one could and could not do with the tools, based on rough informal arguments. And they worked! In every case, mathematicians eventually came along and set the tools on rigorous foundations, and the tools turned out to work in basically the cases a physicist would expect.
So there’s clearly some epistemic techniques here which aren’t captured by Mathematical Proof + The Scientific Method. Physicists were able to figure out correct techniques before the proofs were available. The Scientific Method played a role - physicists could check their results against real-world data - but that’s mostly just checking the answer. The hard part was to figure out which answer to check in the first place, and that involved informal mathematical arguments.
We don’t really have a legible Art of Informal Mathematical Arguments, the way we have a legible Art of Mathematical Proofs or Art of Scientific Method. Informal mathematics clearly played a key role historically in figuring out useful tools, and will likely continue to play a key role in the future, but we don’t have a step-by-step checklist to follow in order to use informal mathematical arguments (the way we do for The Scientific Method), or even a checklist to verify that we’ve applied informal mathematical arguments correctly (the way we can for Mathematical Proofs). If someone says that my informal mathematical argument is wrong, and I can’t either convert it to a formal proof or show some definitive experiment, then I don’t have a clear standard way to argue that it’s correct.
Yet there’s clearly something making informal mathematical arguments correlate quite highly with truth (if imperfectly), because they work well in practice!
The same applies to Fermi estimates. They work remarkably well in practice, yet there’s not a standard step-by-step checklist. There’s not some standard rules to check whether a Fermi estimate is correct. If someone disagrees with my Fermi estimate, there’s not a way for me to establish correctness other than putting in all the work to find more-rigorously-estimated numbers.
The same applies to gears-level models. Plenty of physicists (and engineers, and others in technical fields) have an intuitive sense that a gears-level model is useful and powerful in ways that a black-box model isn’t. But if someone comes along and says that their 50-variable linear regression gives more precise predictions than a model based on the internal structure of the system, and that we really don’t need those gears anyway, I expect most people would not have a strong explanation of why the gears matter. Such explanations do exist (see e.g. the Lucas Critique, or my own writing on the subject), but gears-level models are still in the early stages of becoming legible. As of today, we don’t even have a widely-accepted explanation/definition of what “gears-level” means! For most practitioners, it’s just a vague aesthetic sense, at most.
The really important point to notice here is not any one method, but that there seems to be a bunch of these. Enough that there’s probably more of them which we haven’t even given names to yet. And they seem to come disproportionately from physics. (Or from the kinds of applied mathematicians who are adjacent to physics, and not adjacent to people who write Definition-Theorem-Proof style textbooks and papers.)
So if we want to learn all these key illegible methods, use them ourselves, and maybe someday make them more legible, then physics (and physics-adjacent applied math) seems like the main subject to study.
One caveat to anyone following this advice: once you get past the basics, it is probably more important to study the work of physicists as opposed to physics per se. Even outside of physics itself, there are certain patterns of thought which will make it clear who the physicists are - for instance, I could guess that Uri Alon got his degree in physics, even though he’s known mainly for his introductory text on systems biology. It’s exactly those illegible epistemic tools which identify such people; once you have a little bit of a handle on the tools, you’ll hopefully be able to recognize other people using them. And by studying how those tools are used outside physics itself, we can get a wider view of their application.
I'm wondering about the different types of intuitions in physics and mathematics.
What I remember from prepa (two years after high school where we did the full undergraduate program of maths and physics) was that some people had maths intuition (like me) and some had physics intuition (not me). That's how I recall it, but thinking back on it, there were different types of maths intuitions, which correlated very differently with physics intuition. I had algebra intuition, which means I could often see the way to go about algebraic problems, whereas I didn't have analysis intuition, which was about variations and measures and dynamics. And analysis intuition correlated strongly with physical intuition.
It's also interesting that all your examples of physicist using informal mathematical reasoning successfully ended up being formalized through analysis.
This observation makes me wonder if there are different forms of "informal mathematical reasoning" underlying these intuitions, and how relevant each one is to alignment.
Also the distinction becomes fuzzy because there's a lot of tricks which allow one to use a type of intuition to study the objects of the other type (things like analytic methods and inequalities in discrete maths, let's say, or algebraic geometry). Although maybe this is just evidence that people tend to have one sort of intuition, and want to find way of applying it at everything.
Interestingly, I have better algebra intuition than analysis intuition, within math, and my physics intuition almost feels more closely related to algebra (especially "how to split stuff into parts") than analysis to me.
Although there's another thing which is sort of both algebra and analysis, which is looking at a structure in some limit and figuring out what other structure it looks like. (Lie groups/algebras come to mind.)