You don't get to memorize less this way. You learn from simpler to harder, not from smaller to larger. If you already know all the relevant math (linear algebra, complex analysis, partial differential equations), it might be interesting to start from, say, QM and then derive CM from it. But wait, shouldn't you start even smaller, with QFT, or at least with the Standard Model of Particle Physics, then proceed to peel off QCD and QED, then extract a Hilbert space from the Fock space and do QM, then construct CM and E&M... But that's not enough, what about gravity? Better learn GR, then derive SR from it, and Newtonian gravity as well.

I suppose it's not impossible, but the amount of math you would have to learn before you finally derive that F=ma is rather significant. In some parallel world, where every physicist learns a lot of math first, it might even make sense. But if you want to get some useful results early, and not spend 4 years learning math before you even think about physics, then you should probably start with classical mechanics and classical electrodynamics.