Mathematicians would probably call much less of what physicists do "math" than the physicists. Let me focus on statistical mechanics. A century ago, physicists made assertions that mathematicians could understand, like the central limit theorem and ergodicity. There was debate about whether these were mathematical or physical truths, but it is fine to take them as assumptions and do mathematics. This happens today with spin glasses. But physicists also talk about universality. I suppose that's a precise claim, though rather strong, but the typical prototype of a universality class is a conformal field theory and mathematicians can't make heads or tails of that. The calculations about CFT may look like math, but the rules aren't formal.

PS - bank tellers per capita fell from 1998 to 2008, though not much.