While I do think that many pop-sci explanations of theoretical physics are fairly worthless and often actively misleading, I do not think that it is impossible to gain real insight into (say) the general theory of relativity without mastering differential geometry. Geroch's book presupposes only high school mathematics, but it provides a genuinely deep insight into relativity.

I described the requirements as algebra, analytic geometry, and basic calculus, which is more or less within advanced high-school math. (Without calculus, I don't see how you could explain integrals along the world line, which are the very heart of the matter.)

However, note that just high-school algebra is already worlds apart from purely prose-based, general-audience pop-science. I would guess that for an average reader (let alone owner) of pop-science books, following a text using algebra would be far harder than it would be to figure out tensors for a reasonably math-savvy twelfth grade student.