It might be worthwhile to get to the level of something like "mathematics for electrical engineers", which is reasonably easy and would cover all the math presented or mentioned on this site, including QM. It is unreasonable to shoot for the graduate level in math, unless you actually have the goal of doing a PhD in math, or unless you are a math genius. The difference in effort between the two is at least an order of magnitude (there are many more topics to cover, and each topic is harder than the one before), while the payoff difference is negligible. Well, unless (another unless) your goal is to research AGI/FAI.
Given that, have you thought through the utility of learning all this math vs doing something else equally hard and long, and possibly boring?
This will not be a long post; I have a simple question to ask: if you wanted to educate yourself to graduate level in mathematics, but didn't actually want to go to university, what would you do? I would ask for text-book recommendations, but I don't want to limit your responses (however, bear in mind that the wikipedia articles on, say, cardinality or well-ordering go over my head – they may skim my hairline, but over they go). Also bear in mind that while I personally have A-levels (British qualifications) in both Maths and Further Maths (which is to say, I know some calculus at least), there are probably plenty of people on lesswrong who don't and who desire the same information – so assume as much ignorance as you feel necessary (it's a shame, actually, that there isn't a sequence here on lesswrong for maths). What do you advise (if you think the query ill-defined, I would like to know that as well)?