But it's not obvious to me that the liberalism of some of its supporters is the actual reason for the problems he has with it.

It is in general difficult to tell what the "actual" motivations are for an individual's beliefs. Often they are complicated. Regarding math and physics there's a general pattern that Andrew doesn't like things that are counterintuitive. I suspect that the dislike of special and general relativity comes in part from that.