It turns out there's an extremely straightforward mathematical reason why simplicity is to some extent an indicator of high probability.

Consider the list of all possible hypotheses with finite length. We might imagine there being a labeling of this list, starting with hypothesis 1, then hypothesis 2, and continuing on for an infinite number of hypotheses. This list contains the hypotheses capable of being distinguished by a human brain, input into a computer, having their predictions checked against the others, and other nice properties like that. In order to make predictions about which hypothesis is true, all we have to do is assign a probability to each one.

The obvious answer is just to give every hypotheses equal probability. But since there's an infinite number of these hypotheses, that can't work, because we'd end up giving every hypothesis probability zero! So (and here's where it starts getting Occamian) it turns out that any valid probability assignment has to get smaller and smaller as we go to very high numbers in the list (so that the probabilities can all add up to 1). At low numbers in the list the probability is, in general, allowed to go up and down, but hypotheses with very high numbers always have to be low probability.

There's a caveat, though - the position in the list can be arbitrary, and doesn't have to be based on simplicity. But it turns out that it is impossible to make any ordering of hypotheses at all, without having more complicated hypotheses have higher numbers than simpler hypotheses on average.

There's a general argument for this (there's a more specific argument based on universal turing machines that you can find in a good textbook) that's basically a reflection of the fact that there's a most simple hypothesis, but no "most complex" hypothesis, just like how there's no biggest positive integer. Even if you tried to shuffle up the hypotheses really well, you have to have each simple hypothesis end up at some finite place in the list (otherwise they end up at no place in the list and it's not a valid shuffling). And if the simple hypotheses are all at finite places in the list, that means there's still an infinite number of complex hypotheses with higher numbers, so complexity still decreases for large enough places in the list.

If you have information that simplicity works good in the field of application then the success should be attributed to this information rather than simplicity per se. There are no free lunches and the prior information about the fittness of simple theories is your toll for being able to occam your way forward.

The point is that two theories of incomparable theory branches can't be ordered with occam but one being an elaboration of another (ie up or down the same branch) can. Other uses are better understood as unrelated to this idea but still falsely attributed to it.

My reading of the main argument here is that human theories are more likely when simple, not because of any fact about theory-space but because of the nature of humans' theory-generation process. In particular, theories that aren't correct acquire elaborations to make them give the right predictions, a la epicycles. This requires that theories are more likely if they can make correct predictions without lots of elaborations (or else theories with epicycles would be correct as often as those without). But in order for this rule to differ from Occam's Razor, we need to be able to decide what's the core of a theory and what's 'elaboration', so we can penalize only for the latter. I can't see any way to do this, and the author doesn't offer one either. And when you think in a Turing-Machine type formalism where hypotheses are bit strings representing programs, separating the 'core' of the bit-string-program from the 'elaborations' of the bit-string-program doesn't seem likely to succeed.

You should read up on regularization) and the no free lunch theorem, if you aren't already familiar with them.

A theory is a model for a class of observable phenomena. A model is constructed from smaller primitive (atomic) elements connected together according to certain rules. (Ideally, the model's behavior or structure is isomorphic to that of the class of phenomena it is intended to represent.) We can take this collection of primitive elements, plus the rules for how they can be connected, as a modeling language. Now, depending on which primitives and rules we have selected, it may become more or less difficult to express a model with behavior isomorphic to the original, requiring more or fewer primitive elements. This means that Occam's razor will suggest different models as the simplest alternatives depending on which modeling language we have selected. Minimizing complexity in each modeling language lends a different bias toward certain models and against other models, but those biases can be varied or even reversed by changing the language that was selected. There is consequently nothing mathematically special about simplicity that lends an increased probability of correctness to simpler models.

That said, there are valid reasons to use Occam's razor nonetheless, and not just the reasons the author of this essay lists, such as resource constraint optimization. In fact, it is reasonable to expect that using Occam's razor does increase the probability of correctness, but not for the reasons that simplicity alone is good. Consider the fact that human beings evolved in this environment, and that our minds are therefore tailored by evolution to be good at identifying patterns that are common within it. In other words, the modeling language used for human cognition has been optimized to some degree to easily express patterns that are observable in our environment. Thus, for the specific pairing of the human environment with the modeling language used by human minds, a bias towards simpler models probably is indicative of an increased likelihood of that model being appropriate to the observed class of phenomena, despite simplicity being irrelevant in the general case of any arbitrary pairing of environment and modeling language.

Looking at the machine learning section of the essay, and the paper it mentions, I believe the author to be making a bit too strong a claim based on the data. When he says:

"In some cases the simpler hypotheses were not the best predictors of the out-of-sample data. This is evidence that on real world data series and formal models simplicity is not necessarily truth-indicative."

... he fails to take into account that many more of the complex hypotheses get high error rates than the simpler hypotheses (despite a few of the more complex hypotheses getting the smallest error rates in some cases), which still says that when you have a whole range of hypotheses, you're more likely to get higher error rates when choosing a single complex one than a single simple one. It sounds like he says Occam's Razor is not useful just because the simplest hypothesis isn't ALWAYS the most likely to be true.

Similarly, when he says:

"In a following study on artificial data generated by an ideal fixed 'answer', (Murphy 1995), it was found that a simplicity bias was useful, but only when the 'answer' was also simple. If the answer was complex a bias towards complexity aided the search."

This is not actually relevant to the discussion of whether simple answers are more likely to be fact than complex answers, for a given phenomenon. If you say "It turns out that you're more likely to be wrong with a simple hypothesis when the true answer is complex", this does not affect one way or the other the claim that simple answers may be more common than complex answers, and thus that simple hypotheses may be, all else being equal, more likely to be true than complex hypotheses when both match the observations.

That being said, I am sympathetic to the author's general argument. While complexity (elaboration), when humans are devising theories, tends to just mean more things which can be wrong when further observations are made, this does not necessarily point to whether natural phenomena is generally 'simple' or not. If you observe only a small (not perfectly representative) fraction of the phenomenon, then a simple hypothesis produced at this time is likely to be proven wrong in the end. I'm not sure if this is really an interesting thing to say, however - when talking about the actual phenomena, they are neither really simple nor complex. They have a single true explanation. It's only when humans are trying to establish the explanation based on limited observation that simplicity and complexity come into it.

I notice I am confused. I have updated my understanding of Occam's razor numerous times in the past.

In this case I am helped by wikipedia; " The principle states that among competing hypotheses that predict equally well, the one with the fewest assumptions should be selected. Other, more complicated solutions may ultimately prove to provide better predictions, but—in the absence of differences in predictive ability—the fewer assumptions that are made, the better." https://en.wikipedia.org/wiki/Occam%27s_razor

That statement would suggest that the essay is entirely not about Occham's Razor but about "simple theories".

Whenever I explain Occham today I try to describe. Imagine if you will; a toaster. Bread goes into the "MAGICAL BLACK TOASTER BOX"[1], and toast comes out. As a theory for how a toaster works; its fine; except that it relies on Magic and an assumption of what that is. For practical purposes, knowing that a toaster just works might be more efficient in life than the real explanation of how a toaster works, (electricity passes through coils which heat up due to electrical resistance which cause enough heat to cook a piece of bread and turn it into toasted bread[2]). However if you compare the two explanations; one doesn't really explain how a toaster works; and the other leaves a lot less unexplained. (sure electricity is unexplained but thats a lot less than MAGIC TOASTING BOX). So Occam would suggest that [2] is more likely to be true than [1].

Does this make sense? Have I got my understanding of toasting magic and Occam correct?

From the article.

an inherent bias towards simplicity in the natural world

uhhhh....

Occam's Razor is a heuristic, that is, a convenient rule of thumb that usually gives acceptable results. It is not a law of nature or a theorem of mathematics (no, not an axiom either). It basically tells you what humans are likely to find more useful. It does not tell you -- and does not even pretend to tell you -- what is more likely to be true.