If a researcher was given 1000X more data, 1000X CPU power, would he switch to a brute-force approach? I did not see the connection between "data and computation power" and the brute-force models.

Side note: one topic I've been reading about recently which is directly relevant to some of your examples (e.g. thunderstorms) is multiscale modelling. You might find it interesting.

To take the climate example, say scientists had figured out that there were a biological feedback that kicks in once global warming has gone past 2C (e.g. bacteria become more efficient at decomposing soil and releasing CO2). Suppose you have one model that includes a representation of that feedback (e.g. as a subprocess) and one that does not but is equivalent in every other way (e.g. is coded like the first model but lacks the subprocess). Then isn't the second model simpler according to metrics like the minimum description length, so that it would be weighted higher if we penalised models using such metrics? But this seems the wrong thing to do, if we think the first model is more likely to give a good prediction.

The trick here is that the data on which the model is trained/fit has to include whatever data the scientists used to learn about that feedback loop in the first place. As long as that data is included, the model which accounts for it will have lower minimum description length. (This fits in with a general theme: the minimum-complexity model is simple and general-purpose; the details are learned from the data.)

Now the thought that occurred to me when writing that is that the data the scientists used to deduce the existence of the feedback ought to be accounted for by the models that are used, and this would give low posterior weight to models that don't include the feedback. But doing this in practice seems hard.

... I'm responding as I read. Yup, exactly. As the Bayesians say, we do need to account for all our prior information if we want reliably good results. In practice, this is "hard" in the sense of "it requires significantly more complicated programming", but not in the sense of "it increases the asymptotic computational complexity". The programming is more complicated mainly because the code needs to accept several qualitatively different kinds of data, and custom code is likely needed for hooking up each of them. But that's not a fundamental barrier; it's still the same computational challenges which make the approach impractical.

it's not clear to me if there would be a way to tell between models that represent the process but don't connect it properly to predicting the climate...

Again, we need to include whatever data allowed scientists to connect it to the climate in the first place. (In some cases this is just fundamental physics, in which case it's already in the model.)

If our models were deterministic, then if they were not true, wouldn't it be impossible for them to produce the observed data exactly, so that the likelihood of the data given any of those models would be zero? (Unless there was more than one process that could give rise to the same data, which seems unlikely in practice.)

Picture a deterministic model which uses fundamental physics, and models the joint distribution of position and momentum of every atom comprising the Earth. The unknown in this model is the initial conditions - the initial position and momentum of every particle (also particle identity, i.e. which element/isotope each is, but we'll ignore that). Now, imagine how many of the possible initial conditions are compatible with any particular high-level data we observe. It's a massive number!

Point is: the deterministic part of a model of a fundamental physical model is the dynamics; the initial conditions are still generally unknown. Conceptually, when we fit the data, we're mostly looking for initial conditions which match. So zero likelihoods aren't really an issue; the issue is computing with a joint distribution over position and momentum of so many particles. That's what statistical mechanics is for.

whilst statistical averaging has got human modellers a certain distance, adding representations of processes whose effects get missed by the averaging seems to add a lot of value

The corresponding problem in statistical mechanics is to identify the "state variables" - the low-level variables whose averages correspond to macroscopic observables. For instance, the ideal gas law uses density, kinetic energy, and force on container surfaces (whose macroscopic averages correspond to density, temperature, and pressure). Fluid flow, rather than averaging over the whole system, uses density and particle velocity within each little cell of space.

The point: if an effect is "missed by averaging", that's usually not inherent to averaging as a technique. The problem is that people average over poorly-chosen features.

Jaynes argued that the key to choosing high-level features is reproducibility: what high-level variables do experimenters need to control in order to get a consistent result distribution? If we consistently get the same results without holding X constant (where X includes e.g. initial conditions of every particle), then apparently X isn't actually relevant to the result, so we can average out X. Also note that there's some degrees of freedom in what "results" we're interested in. For instance, turbulence has macroscopic behavior which depends on low-level initial conditions, but the long-term time average of forces from a turbulent flow usually doesn't depend on low-level initial conditions - and for engineering purposes, it's often that time average which we actually care about.

if the models we can afford to compute with can't reproduce the data, then presumably they are also not reproducing the correct causal graph exactly? And any causal graph we could compute with will not be able to reproduce the data?

Once we move away from stat mech and approximations of low-level models, yes, this becomes a problem. However, two counterpoints. First, this is the sort of problem where the output says "well, the best model is one with like a gazillion edges, and there's a bunch that all fit about equally well, so we have no idea what will happen going forward". That's unsatisfying, but at least it's not wrong. Second, if we do get that sort of result, then it probably just isn't possible to do better with the high-level variables chosen. Going back to reproducibility and selection of high-level variables: if we've omitted some high-level variable which really does impact the results we're interested in, then "we have no idea what will happen going forward" really is the right answer.

Extreme answer: just point AIXI at wikipedia. That's a bit tongue-in-cheek, but it illustrates the concepts well. The actual models (i.e. AIXI) can be very general and compact; rather than AIXI, a specification of low-level physics would be a more realistic model to use for climate. Most of the complexity of the system is then learned from data - i.e. historical weather data, a topo map of the Earth, composition of air/soil/water samples, etc. An exact Bayesian update of a low-level physical model on all that data should be quite sufficient to get a solid climate model; it wouldn't even take an unrealistic amount of data (data already available online would likely suffice). The problem is that we can't efficiently compute that update, or efficiently represent the updated model - we're talking about a joint distribution over positions and momenta of every particle comprising the Earth, and that's even before we account for quantum. But the prior distribution over positions and momenta of every particle we can represent easily - just use something maxentropic, and the data will be enough to figure out the (relevant parts of the) rest.

So to answer your specific questions:

• the "true model space" is just low-level physics
• by "efficiently", I mean the code would be writable by a human and the "training" data would easily fit on your hard drive