It seems to me that in science, there is always an implicit agreement that the current theory could be revised in light of new contradictory evidence. As far as I can tell, the Bayesian approach seems to lack this feature, since we have to assume a fixed model of the world to do the probability updates.
For example, what's the probability that the sun will rise tomorrow? How do you even calculate it? (To keep things simple, suppose you have seen the sun rise N times). More abstractly, suppose every day you get a bit of information from some source, and the first N bits are all one. What's the probability that the next bit is one? How would the perfect Bayesian mind answer that?
An interesting way to avoid all this is to simply look at behavior (rather than beliefs) and apply an evolutionary argument which goes like this: Finding and exploiting patterns is useful for survival, so evolution favored organisms that could do so. No "laws" required. The universe just needs to be orderly enough for life to survive. It need not make sense all the way down. I don't believe it, but it's interesting nevertheless.
New Scientist on changing the definition of science, ungated here:
I'm a good deal less of a lonely iconoclast than I seem. Maybe it's just the way I talk.
The points of departure between myself and mainstream let's-reformulate-Science-as-Bayesianism is that:
(1) I'm not in academia and can censor myself a lot less when it comes to saying "extreme" things that others might well already be thinking.
(2) I think that just teaching probability theory won't be nearly enough. We'll have to synthesize lessons from multiple sciences like cognitive biases and social psychology, forming a new coherent Art of Bayescraft, before we are actually going to do any better in the real world than modern science. Science tolerates errors, Bayescraft does not. Nobel laureate Robert Aumann, who first proved that Bayesians with the same priors cannot agree to disagree, is a believing Orthodox Jew. Probability theory alone won't do the trick, when it comes to really teaching scientists. This is my primary point of departure, and it is not something I've seen suggested elsewhere.
(3) I think it is possible to do better in the real world. In the extreme case, a Bayesian superintelligence could use enormously less sensory information than a human scientist to come to correct conclusions. First time you ever see an apple fall down, you observe the position goes as the square of time, invent calculus, generalize Newton's Laws... and see that Newton's Laws involve action at a distance, look for alternative explanations with increased locality, invent relativistic covariance around a hypothetical speed limit, and consider that General Relativity might be worth testing. Humans do not process evidence efficiently—our minds are so noisy that it requires orders of magnitude more extra evidence to set us back on track after we derail. Our collective, academia, is even slower.