Yes, there are a lot of constraints. They aren't all hard constraints, but that's life. I'm not going to put all my eggs into one basket, but I think I would be about as significantly unhappy if I can't resolve career issues as if I can't live within a 4-hour plane ride radius from my family.
Regarding the part-time job thing: it would be pretty hard to sustain myself financially that way. Like I said, I have additional goals like saving money for retirement/home ownership. I don't see this as being orthogonal to doing a career I derive personal meaning from at all: being able to do this is a standard byproduct from most careers at the earnings levels that my master's degree would provide. How do I pinpoint the subset of those careers that also satisfy the constraint that they pay me money to do mathematical research?
I've posted a few things seeking career advice with mixed success. In this case I have a more concrete question and if you feel like commenting, I'd appreciate it. I think it helps me to hear what a community of others thinks from a rational perspective because there are often many components to a decision that I had not anticipated.
I am currently a grad student working in computer vision. I dislike the way that my current adviser focuses only on projects that have short-term commercial gains. I want to study more fundamental, theoretical research which may take more time to develop but will also be more aesthetically pleasing to me. For me, the only reason to agree to be paid so little as a graduate student is to gain the opportunity to work freely on high risk projects that happen to be of personal interest. Practical considerations are not interesting to me as motivation for a Ph.D. On the other hand, it has felt nearly impossible to actually find faculty willing to have students work on theory. Rather than grinding away with no dental insurance for 3 more years, followed by low paying post-docs, etc., perhaps seeking a job will be better.
I have some interesting job prospects that are all with larger companies. The jobs are basically business analytics, including scientific computing, data mining, and machine learning. I'm sure the problems to work on are not that great; not going to be Earth shattering, but at the same time they sound a lot more interesting to me than hedge fund data analysis or military research labs (I have working experience at a government lab and I did not enjoy it). The hours would be better; the pay is fair and it would be a good living. I could pursue some things as serious hobbies outside work.
At the same time though, there feels like a nagging opportunity cost. I am not naive enough to believe there will be a nice faculty job waiting for me even if I finish my Ph.D. However, I really enjoy theoretical and mathematical physics, machine learning, computational complexity, and scientific computing, and various philosophical considerations generated by these. Being able to teach about them, research them, and work on them professionally seems incredibly appealing. Am I making a big mistake if I leave? How can one pursue philosophical interests and desires to work in theory outside of a typical job? Or should I even worry about such a thing?