His thoughts on that are confused too. He claims that math is fundamental to physics, but also that it's infinitely complex. That doesn't work:

1) Math is simple in the sense that you need very little space to specify the entities needed to use it.

2) But Landsburg says it's complex because you haven't really specified it until you know every mathematical truth.

3) But then physics isn't using math by that definition! It's using a tiny, computable, non-complex subset of that.

(This is discussed at length in the links I gave.)

Thought-provoking is good, but don't fall for the trap of worshipping someone for saying stuff that doesn't make sense.