When I was choosing between math and physics as a kid, I felt that they are similar in the way that matters, which is having exact results that stay true everywhere and for all time. And they felt similar in terms of mental effort and enjoyment as well. Mostly I just wanted to do something I would like, I didn't care about applicability, and understood that there wouldn't be much applicability either way. I ended up choosing math, my brother and my parents went into physics, and we all sort of gravitated toward computers eventually.

It's true that ideas from physics seem less applicable to everyday life than ideas from math, but IMO the important part is the style of thinking, not the ideas themselves. Seeing how many physics graduates go to finance, it seems that physics is at least as good a "mental sport" as math.