Hey, Math PhD candidate here (graduating this May).

Long-term, my career goals are to retire early (I've felt comfortable enough on what I live on as a student that the MrMoneyMustache approach seems eminently doable)...

with the actual terminal values involved being enjoyment and lack of stress....

These are my goals, as well.

Teaching just sounds horrible to me.

It is pretty horrible. My university has a relatively teaching-heavy TA assignment, and it was kind of soul-crushing.

becoming a quant also seems like a bad choice what with having to get a PhD first.

Graduate school could serve the role of a holding pattern for you to figure out what it is you actually want to do. I think it's possible to become a quant or an actuary with a MSF or other master's degree. However, I don't recommend going into debt for graduate school, and as far as I know most graduate schools don't fund master's students.

There's a somewhat sneaky trick: One can apply for a PhD program, obtain a TA or RA, and then after the requirements of the Master's program you actually want are done, transfer to the Master's program and graduate out.

Of course that all requires some degree of teaching, probably, and afterwards you need to find a job making enough to balance out the cost of making a TA's salary for ~2 years.

The people I know who retired or are scheduled to retire the quickest do white-collar jobs in manufacturing or energy at very large corporations.

The people I know who do the least stressful jobs work either part-time in retail or have tenure of one kind or another. One is an epic-level computer programmer who gets so many job offers that he's able to choose the least restrictive.

Me, personally? After I defend I'm going to work for a small research lab.