In full context of the paper, Gelman is noting this as a problem with standard Bayesian analysis. He doesn't argue, as I'm arguing, that we're trying to model our priors or the structure of our uncertainty, i.e. that we're trying to approximate the fully Baysian answer.
After going back and re-reading this, I realized your comments are more prescient than I gave them credit for in the past. I'm now struggling with the Gelman-Shalizi article (link). Do you know of any LessWrong sources that discuss this. I need to really sit back and think, but it seems to me that Gelman and Shalizi are making some serious mistakes here. And they are two of the best practitioners I know of. That scares me a great deal.
Andrew Gelman recently linked a new article entitled "Induction and Deduction in Bayesian Data Analysis." At his blog, he also described some of the comments made by reviewers and his rebuttle/discussion to those comments. It is interesting that he departs significantly from the common induction-based view of Bayesian approaches. As a practitioner myself, I am happiest about the discussion on model checking -- something one can definitely do in the Bayesian framework but which almost no one does. Model checking is to Bayesian data analysis as unit testing is to software engineering.
Added 03/11/12
Gelman has a new blog post today discussing another reaction to his paper and giving some additional details. Notably: