It's worse than that, even. What math textbooks you can pick up and get something out of is determined by your current math background, so it's also a huge element of luck to get books that give you progress instead of discouragement.

I see it as more of a "maybe there is low-hanging fruit and you can teach yourself some math easily, but good luck picking topics you can tackle"