I understand that objection, but I disagree. We do have at least two pieces of relevant evidence:

1. The universe we find ourselves in seems to be a rather simple mathematical object. The final verdict on this, of course, is still out; but if the bottom were something extremely complicated, it would seem to be an unbelievable coincidence that the approximations we've found (QM and GR) are so tidy and so very very accurate.

2. We can usually think of good reasons that simpler mathematical objects wouldn't contain a great proportion of observers. The particular cosmology of our universe appears to be one of the simplest ways to make evolution of sentient life plausible— for example, QM without gravity results in a dispersive wavefunction, and remotely brain-sized interacting configurations become vanishingly unlikely.

I mean, you could build a conscious agent in Conway's Game of Life, but it seems unlikely that any not-too-complicated starting configuration would result in one (well, a chaotic one would result in very occasional Boltzmann brains, but those would be much rarer than conscious life is in our universe, more than enough of a difference to outweigh a reasonable complexity penalty on universes).

From these facts, if we expect to be "typical" elements of our reference class (i.e. conscious agents, perhaps with some additional conditions), something like the Level IV Multiverse is strongly supported, and predicts that 1 and 2 will continue to be more strongly validated. (2, in particular, is a pretty decent prediction; it says that we can't come up with a simpler cosmology-leading-to-sustained-evolution than the one we live in).