if you wanted to educate yourself to graduate level in mathematics, but didn't actually want to go to university, what would you do?

If you want to really learn the stuff, and you're not outlandishly talented, it'd still take you at least a couple of years of full-time effort, likely more (and correspondingly even more in part-time effort), assuming you don't trip up and go in a wrong direction (e.g. form shallow understanding of advanced topics instead of building reliable foundation). This is unlikely to be achieved if you aren't led step-by-step by a college system, don't enjoy the process, and don't have something to protect.