what if the funding for research in a field is limited, and there is someone who is better at doing research than you are, but worse at doing job interviews, so by deciding to go into that field, you take their place?
Even in the worst case, I suspect I am much better at doing research than the marginal person who would lose their job. Maybe there are some weird other issues with imperfect candidate selection, but I don't see why those would have any effect on average.
In reality, I expect the situation to be much better, for two reasons. For one, even holding the total amount of research funding fixed, how do you think that the allocation between various fields or subfields is done? I believe the number of competent researchers/grant-writers working in a field effects the fraction of funding it captures. For two, most people in most fields pursue programs which are optimized for maintaining status, not for doing good. You shouldn't expect the amount of socially optimal research to be conserved unless funding agencies are very good. In practice they actually seem to be astoundingly bad (at least for this way of evaluating good/bad). I guess these points are the same, but at different levels: working on the best programs within the best subfields of the best fields is important, and I don't think the average person you statistically expect to replace is doing any of these.
if there's a profitable niche, it will be filled by someone sooner or later
I believe this for a sufficiently generous notion of "later," and for enterprises which create marketable value proportionate with the good they do. But I care about shorter timescales than niches generically get filled on, and my notion of value is quite different than most peoples. Evidence for the first point comes from the amount of money that goes to successful entrepreneurs. Evidence for the second comes from looking around at the world, or reasonable expectations of marketplaces.
In both cases, I think others' remarks are essentially correct: markets work to distribute labor evenly in some sense, but my motives are different from my competitors' and so I suspect I will displace people who are doing substantially different things.
I found out that if I work in some field I'll get bored after a time-frame of months to years, and this was not something I expected when I was in college.
I guess I don't know if this will be a problem. I have definitely gotten bored of particular problems, but at MIT at least there are a pretty wide variety of problems being worked on and it seems pretty easy to move around. Maybe it will become harder to move around (I imagine I can avoid this) or a higher level of meta-boredom will set in which would require a more disruptive change.
Right now this is not high on my list of reasons not to go into academia, though it would be good to know more before spending years getting a PhD.
Another thing to check is how much you like writing and publishing academic papers. I tried it once and found the process quite painful, both the writing and the publishing parts. (And that was one of the actual reasons I didn't try to go into academia. I didn't find out about the boredom issue until later.) I'm not sure if I'm being rational or just rationalizing, but it seems that I can spread my ideas (and get plenty of credit) just by writing about them informally on mailing lists and blogs.
I have been having some difficulty deciding what to do with my life. I'm not really ashamed or surprised, because the problem seems extremely difficult and worth getting right. I still don't really know. Here are some of my options, though I wouldn't be surprised if what I end up doing is not on the list. I thought I would share to elicit advice, to give some context to some of my recent remarks, and maybe to provide comparison for people in similar situations.
1. Research
I could do research in a university, or in a private research lab. Many fields have at least a few questions that seem important and interesting.
A. Theoretical Computer Science. I could work on collaborative learning, recommendation and reputation systems, distributed protocols, or anything else that might assist collaboration in the future.
B. AI / Computational Cognitive Neuroscience. I could work on algorithms for inference and planning, to increase the probability that we develop comprehensible or human-like AI before something horrible happens (like developing incomprehensible strong AI).
C. Neuroscience. I know almost nothing about this field, but technology for measuring and interfacing with the brain appears to be important and developing rapidly. I don't know how hard it would be to start working in the field, but I suspect that the connection to computer science is strong enough at MIT that it wouldn't be impossible.
2. Start a company
This seems way harder, but I am sufficiently arrogant and risk-neutral that I consider it a reasonable option. In particular, this is what I would do if I decided that making money as efficiently as possible and giving it away
A. Tech company. If I had to guess based on the currently available evidence, I would guess that this is the way to maximize my expected earnings.
B. Online Education. I would like to take a shot at designing materials for online education of smart, significantly underserved, high school students.
3. Cooperate with Other Rationalists
A. Work for a rational charity. Self-explanatory. Probably worse than earning a lot of money and giving away.
B. Start a rational charity. Probably worse than supporting an existing charity.
C. Work for a rational start-up. Can't really arrange this one; but optimistically it could happen if you were prepared (someone else does 2).
4. Be a Hobbyist
I could also simply try and earn a living as quickly as possible (rather than making as much excess as possible, or having a more structured way of doing good) and do work on the side. I don't think this is a good idea.