To the extent that this is true, perhaps the very notion of an epistemic rationalist (perhaps also of epistemic rationality) is incoherent. ("Epistemic rationality means acting so as to maximize one's accuracy." "Ah, but hidden in that word accuracy is some sort of evaluation, which you aren't allowed to have.") But it sure seems like a useful notion.

I propose that there is at least one useful notion of epistemic rationality; in fact, there's one for each viable notion of what counts as better accuracy; since real people have utility functions, calling a real person an epistemic rationalist is really shorthand for "has a utility function that highly values accuracy-in-some-particular-sense"; that one can usefully talk about epistemic rationality in general, meaning something like "things that are true about anyone who's an epistemic rationalist in any of that term's many specific senses"; and that it's at least a defensible claim that something enough like K-L divergence to make Peter's argument go through is likely to be part of any viable notion of accuracy.

If epistemic rationalists can't speak of a "better approximation," then how can an epistemic rationalist exist in a universe with finite computational resources?

This is basically right, but I guess I think of it in slightly different terms. The KL divergence embodies a particular, implicit utility function, which just happens to be wrong lots of the time. So it can make sense to speak of "better_KL", it's just not something that's necessarily very useful.

Note also that alternative divergence measures, embodying different implicit utility functions, could give different answers. For example, Jensen-Shannon divergence would agree with instrumental rationality here, no? (Though you could obviously construct examples where it too would diverge from our actual utility functions.)