Keep in mind that he and other physicists do not generally consider "probability that it is noise, given an observation X" to even be a statement about the world (it's a statement about one's personal beliefs, after all, one's confidence in the engineering of an experimental apparatus, and so on and so forth)
It's about the probability that there is an effect which will cause this deviation from background to become more and more supported by additional data rather than simply regress to the mean (or with your wording, the other way around). That seems fairly based-in-the-world to me.
The actual reality either has this effect, or it does not. You can quantify your uncertainty with a number, that would require you to assign some a-priori probability, which you'll have to choose arbitrarily.
You can contrast this to a die roll which scrambles initial phase space, mapping (approximately but very close to) 1/6 of any physically small region of it to each number on the die, the 1/6 being an objective property of how symmetrical dies bounce.
Another month has passed and here is a new rationality quotes thread. The usual rules are: