When you say "I have really thought about this a considerable amount", I hear "I have diagnosed the problem quite a while ago and it's creating a pit in my stomach but I haven't taken any action yet". I can't give you any points for that.
When you're dealing with a difficult problem and if you're an introspective person it's easy to get stuck in a loop where you keep going through the same sorts of thoughts. You realize you're not making much progress but the problem remains so you feel obligated to think about it some more. You should think more, right? It's an important decision after all?
Nope. Thinking about the problem is not a terminal goal. Thinking is only useful insofar it leads to action. And if your thinking to action ratio is bad, you'll get mentally exhausted and you'll have nothing to show for it. It leads to paralysis where all you do is think and think and think.
If you want to make progress you have to find a way to decompose your problem into actionable parts. Not only will action make you feel better, it's also going to lead to unexplored territory.
So what kind of actions can you take?
Well, your claim is that major conferences require short term commercial papers. So if you go systematically through the papers published in the last year or so you'll find either (a) all the papers are boring, stupid, silly or wrong. (b) there are a bunch of really cool papers in there. In case of (a) maybe you're in the wrong field of research. Maybe you should go into algorithms or formal semantics. In this case look at other computer science papers until you find papers that do excite you. In case of (b) contact the authors of the papers; check out their departments; etc, etc.
To recap: Find interesting papers. Find departments where those interesting papers were written. Contact those departments.
Another strategy. Go to the department library and browse through random books that catch your eye. This is guaranteed to give you inspiration.
This is just from the top of my head. But whatever you do, make sure that you don't just get stuck in a circle of self-destructive thought. Action is key.
If you're certain you want to eventually get a faculty job, do a combination of teaching and research, own a house and regularly go on holiday, then I can't think of any alternatives to the conventional PhD -> faculty route. What's the best way to achieve a faculty job? I don't know. Probably a combination of networking, people skills and doing great research. If you want a faculty job badly enough you can get one. But once you get it there's no guarantee you're going to be happy if what you really want is complete autonomy.
I'm sorry I can't give any targeted advice.
(PS: some people like the idea of travel more than they like travel and some people like the idea of home-ownership more than they like home-ownership. For instance, if you haven't traveled a lot in the past 5 years you probably don't find travel all that important (otherwise you would've found a way to travel).)
If you want a faculty job badly enough you can get one.
I disagree. I think much of the evidence about the rise of post-docs as principal investigators and the diminishing number of tenured positions is at odds with this claim. This claim is essentially why most students go to a Ph.D. program and they become depressed when they learn it doesn't work like this about 3 years into the process.
For instance, if you haven't traveled a lot in the past 5 years you probably don't find travel all that important (otherwise you would've found a way to travel).
In...
I've posted a few things seeking career advice with mixed success. In this case I have a more concrete question and if you feel like commenting, I'd appreciate it. I think it helps me to hear what a community of others thinks from a rational perspective because there are often many components to a decision that I had not anticipated.
I am currently a grad student working in computer vision. I dislike the way that my current adviser focuses only on projects that have short-term commercial gains. I want to study more fundamental, theoretical research which may take more time to develop but will also be more aesthetically pleasing to me. For me, the only reason to agree to be paid so little as a graduate student is to gain the opportunity to work freely on high risk projects that happen to be of personal interest. Practical considerations are not interesting to me as motivation for a Ph.D. On the other hand, it has felt nearly impossible to actually find faculty willing to have students work on theory. Rather than grinding away with no dental insurance for 3 more years, followed by low paying post-docs, etc., perhaps seeking a job will be better.
I have some interesting job prospects that are all with larger companies. The jobs are basically business analytics, including scientific computing, data mining, and machine learning. I'm sure the problems to work on are not that great; not going to be Earth shattering, but at the same time they sound a lot more interesting to me than hedge fund data analysis or military research labs (I have working experience at a government lab and I did not enjoy it). The hours would be better; the pay is fair and it would be a good living. I could pursue some things as serious hobbies outside work.
At the same time though, there feels like a nagging opportunity cost. I am not naive enough to believe there will be a nice faculty job waiting for me even if I finish my Ph.D. However, I really enjoy theoretical and mathematical physics, machine learning, computational complexity, and scientific computing, and various philosophical considerations generated by these. Being able to teach about them, research them, and work on them professionally seems incredibly appealing. Am I making a big mistake if I leave? How can one pursue philosophical interests and desires to work in theory outside of a typical job? Or should I even worry about such a thing?