What I think distinguishes physics from mathematics is that the diminishing returns from physics start setting in earlier than they do for mathematics, and the extent of applicability of physics is more limited (for instance, classical mechanics is somewhat useful, but not as much as calculus -- and both are done at roughly the same educational stage).

This doesn't seem obviously true to me. It seems like learning to model actual systems with differential equations and the like would be much more applicable than most upper level mathematics. And the math stuff that is more generally useful, like linear algebra, gets adequately covered in physics. (For what it's worth, I'm a number theory grad student, and I minored in physics as an undergrad).