I guess it depends on what you mean by 'understanding'. I personally feel that you haven't really grasped the math if you've never used it to solve an actual problem - textbook will do, but ideally something not designed for solvability. There's a certain hard-to-convey Fingerspitzggefühl, intuition, feel-for-the-problem-domain - whatever you want to call it - that comes only with long practice. It's similar to debugging computer programs, which is a somewhat separate skill from writing them; I talk about it in some detail in this podcast and these slides.

That said, I would say you can get quite a good overview without any math; you can understand physics in the same sense I understand evolutionary biology - I know the basic principles but not the details that make up the daily work of scientists in the field.