Unless you lack aptitude for math (did you do well in high school?), it looks like you just need to fill the gaps in your basic math education and practice what you learn enough to achieve fluency. Consider walking into your nearest Kumon center and getting an evaluation, then taking your time working through their worksheets, starting from the assessed level. Their program is very instrumental, it is designed to fill the gaps and to build fluency.

As for physics, in my experience most of the trouble people have with freshman physics is due to insufficient math training, though there is an occasional person who just "doesn't get" physics.