(As usual, I have somewhat less extreme views here than Eliezer.)

saying that some analogous but more complicated problem might arise for probabilistic agents

There is a problem here, we have an impossibility proof for a broad class of agents, and we know of no agents that don't have the problem. Indeed, this limits the relevance of the impossibility proof, but it doesn't limit the realness of the problem.

If you have an example of a complication that you think would plausibly arise in practice and have further thoughts on why we shouldn't expect this complication to be avoided by default in the course of the ordinary development of AI, I would be interested in hearing more.

I don't quite see where you are coming from here. It seems like the situation is:

1. There are problems that reflective reasoners would be expected to solve, which we don't understand how to resolve in current frameworks for general reasoning (of which mathematical logic is the strongest).
2. If you think that reflective reasoning may be an important part of AGI, then having formal frameworks for reflective reasoning is an important part of having formal frameworks for AGI.
3. If you think that having formal frameworks is likely to improve our understanding of AGI, then having formal frameworks that support reflective reasoning is a useful step towards improving our understanding of AGI.

The sort of complication I imagine is: it is possible to build powerful AGI without having good frameworks for understanding its behavior, and then people do that. It seems like all things equal understanding a system is useful, not only for building it but also for having reasonable expectations about its behavior (which is in turn useful for making further preparations, solving safety problems, etc.). To the extent that understanding things at a deep level ends up being necessary to building them at all, then what we're doing won't matter (except insofar as people who care about safety making modest technical contributions is indirectly useful).

Do you think that people building AGI in the future will fail to do this research, if it is in fact necessary for building well-functioning AIs? If so, why?

Same answer. It may be that understanding reasoning well is necessary to building powerful agents (indeed, that would be my mode guess). But it may be that you can influence the relative development of understanding vs. building, in which case pushing on understanding has a predictable effect. For example, if people didn't know what proofs or probabilities were, it isn't out of the question that they could build deep belief nets by empirical experimentation. But I feel safe saying that understanding proof and probability helps you better reason about the behavior of extremely powerful deep belief nets.

Here's a stab at how the arguments are different (first thing that came to mind):

I agree that the cases differ in many ways. But this distinction doesn't seem to get at the important thing. To someone working on logic you would say "I don't know whether deduction systems will be formalized in the future, but I know that agents will be able to reason. So this suggests to me that your particular approach for defining reasoning, via formalization, is unnecessary." In some sense this is true---if I'm an early mathematician and I don't do logic, someone else will---but it has relatively little bearing on whether logic is likely to be mathematically productive to work on. If the question is about impact rather than productivity as a research program, then see the discussion above.