I raise (at least) two different related points in my post:

1. "When an argument seems very likely to be wrong but could be right with non-negligible probability, classify it as such, rather than classifying it as false." I think that you're pretty good on this point, and better than I had been.

2. The other is one that you didn't mention in your comment, and one that I believe that you and I have both largely missed in the past. This is that one doesn't need a relatively strong argument to be confident in a subtle judgment call — all that one needs is ~4-8 independent weak arguments. (Note that generating and keeping track of these isn't computationally infeasible.) This is a very crucial point, as it opens up the possibility of no longer needing to rely on single relatively strong arguments that aren't actually too strong.

I believe that the point in #2 is closely related to what people call "common sense" or "horse sense" or "physicists' intuition." In the past, I had thought that "common sense" meant, specifically, "don't deviate too much from conventional wisdom, because views that are far from mainstream are usually wrong." Now I realize that it refers to something quite a bit deeper, and not specifically about conventional wisdom.

I'd suggest talking about these things with miruto.

On many questions in physics, ecology, etc., there's a single factor that dominates all the rest. Maybe this is less true in human domains because rational agents tend to produce efficiencies due to eating up the free lunches.

Our chauffeur from last weekend has recently been telling to me that physicists generally use the "many weak arguments" approach.

For example, the math used in quantum field theory remains without a rigorous foundation, and its discovery was analogous to Euler's heuristic reasoning concerning the product formula for the sine function.

He also referred to scenarios in which (roughly speaking) you have a physical system with many undetermined parameters, and you have ways of bounding different collections of them and that by looking at all of the resulting bounds, you can bound the individual parameters sufficiently tightly so that the whole model is accurate.