But if you live in a world whose structure is Bayesian (as, for example, in our world)?
This isn't as clear, hence why there is still a debate over frequentism vs Bayes. There is also a reason frequentism is usually taught much earlier: it's simpler for children to understand. The introduction to probability theory invariably involves the coin flip or the card draw, where it's simple to assert the existence of a true that can be estimated through repeated trials, which the child can physically do. This view of probability is usually carried on into life without much challenge, in the absence of formal teaching of Bayesian methods.
So it becomes less clear if our world is truly "Bayesian" in structure, since the invocation of Bayesian methodology (the assigning of a prior probability and is subsequent updating) is rarely used by the majority of society whose day-to-day lives rarely require such formalism.
Newtonian gravity is a limiting case of GR (specifically in the low velocity and weak gravity case), which means GR is a strict superset of Newtonian gravity. But it's not accurate to think of frequentism as a limiting case of Bayesian theory. They have two completely different philosophies, and both require assumptions that are sometimes incompatible with each other. The obvious example is the frequentist critique that the choice of prior is arbitrary (ie there being no true uninformative prior).
I also don't think you can assert that the world is inherently Bayesian, given that statistical frameworks exist to interpret the world which has inherent uncertainties. It doesn't make sense to say that there is one true statistical framework under which probabilistic events occur.