Gelman quotes from wikipedia:

Bayesian inference uses aspects of the scientific method, which involves collecting evidence that is meant to be consistent or inconsistent with a given hypothesis. As evidence accumulates, the degree of belief in a hypothesis ought to change. With enough evidence, it should become very high or very low. . . . Bayesian inference uses a numerical estimate of the degree of belief in a hypothesis before evidence has been observed and calculates a numerical estimate of the degree of belief in the hypothesis after evidence has been observed. . . . Bayesian inference usually relies on degrees of belief, or subjective probabilities, in the induction process and does not necessarily claim to provide an objective method of induction.

He then writes:

This does not describe what I do in my applied work. I do go through models, sometimes starting with something simple and building up from there, other times starting with my first guess at a full model and then trimming it down until I can understand it in the context of data. And in any reasonably large problem I will at some point discard a model and replace it with something new (see Gelman and Shalizi 2011a,b, for more detailed discussion of this process and how it roughly fits in to the philosophies of Popper and Kuhn). But I do not make these decisions on altering, rejecting, and expanding models based on the posterior probability that a model is true. Rather, knowing ahead of time that my assumptions are false, I abandon a model when a new model allows me to incorporate new data or to fit existing data better.

I don't disagree with Gelman's statistical practice, but I disagree with his justification. Statistical models are models of our uncertainty about a particular problem. Model checks are a great way to check how well the model is actually modeling our uncertainty and building models up in the fashion Gelman suggests are great ways to find a reasonable model for our uncertainty. Posterior model probabilities - when they can be calculated - are one way of assessing when we have found a better model, but they aren't the only way (and aren't necessarily the best way either).

If we 100% knew our priors (both the prior distribution and the rule by which we update on our prior disttribution, i.e. the likelihood) then Gelman's methods would be useless. Just do the Bayesian update! But we don't actually know our priors, so we must take care to model them as accurately as we can and Gelman's methods are pretty good at helping us do this.