My impression is that a primary difference between the two is that the mathematician learns to be picky, and the physicist learns to not be picky (at least, when it comes to applying mathematical structure to reality). In physics, you learn lots of sophisticated mathematics which you apply without fully checking that it can apply, or learning the underlying assumptions for, because reality does actually exist, which is convenient.