"But what is a "reasonable" distribution? Why not label "reasonable" a very complicated prior distribution, which makes Occam's Razor work in all observed tests so far, but generates exceptions in future cases?"
Occam's Razor is only relevant to model selection problems. A complicated prior distribution does not matter. What does matter is how much the prior distribution volume in parameter space decreases as the model becomes more complex (more parameters). Each additional parameter in the model spreads the prior distribution over an increased parameter space.
For a more complex model to have a higher posterior distribution, the evidence (likelihood) must increase the posterior volume more than the addition prior parameter(s) decrease it.... (read more)
"But what is a "reasonable" distribution? Why not label "reasonable" a very complicated prior distribution, which makes Occam's Razor work in all observed tests so far, but generates exceptions in future cases?"
Occam's Razor is only relevant to model selection problems. A complicated prior distribution does not matter. What does matter is how much the prior distribution volume in parameter space decreases as the model becomes more complex (more parameters). Each additional parameter in the model spreads the prior distribution over an increased parameter space.
For a more complex model to have a higher posterior distribution, the evidence (likelihood) must increase the posterior volume more than the addition prior parameter(s) decrease it.... (read more)