I can address the other questions later on, but I am actually interested in looking to complexity limits for FAI problems. My initial reaction to Yudkowsky's post about cohesive extrapolated volition was that such a thing is probably not efficiently computable, and even if it is, it is probably not stable (in the control theory sense; i.e. a tiny error in CEV yields a disastrously large error in terms of the eventual outcome). It isn't like there is just one single time that we have to have a mathematically comprehensible description of volition. As computational resources grow, I imagine the problem of CEV will be faced many times in a row on rapidly larger scales, and I'm interested in knowing how a reasonable CEV computation scales asymptotically in the size of the projected future generation's computing capabilities. Very very naively, for example, let's say that the number of processors N of some future AI system plays a major role in the mathematical structure of my description of my volition that I need to be prepared to hand to it to convince it to help me along (I know this is a shortsighted way of looking at it, but it illustrates the point). How does the calculation of CEV grow with N. If computing the CEV in a mathematically comprehensible way grows faster than my compute power, then even if I can create the initial CEV, somewhere down the chain I won't be able to. Similarly, if CEV is viewed as a set of control instructions, then above all it has to be stable. If mis-specifying CEV by a tiny percentage yields a dramatically bad outcome, then the whole problem of friendliness may itself be moot. It may be intrinsically unstable.
As far as "math teaching at a respected research university" goes, there are a few reasons. I have a high aesthetic preference for both mathematics and the human light-bulb-going-off effect when students overcome mathematical difficulties, so the job feels very rewarding to me without needing to offer me much in the way of money. I enjoy creating tools that can be used constructively to accomplish things, but I don't enjoy being confined to a desk and needing to focus on a computer screen. The most rewarding experience I have found along these lines is developing novel applied mathematical tools that can then be leveraged by engineers and scientists who have less aversion to code writing. Moreover, I have found that I function much better in environments where there is a vigorous pace to publishing work. At slower places, I tend to chameleonize and become slower myself, but at vibrant, fast-paced places, I seem to function on all cylinders, so to speak. This is why a "respected research university" is much more appealing than a community college or smaller state level college.
I'm very disillusioned with the incentive scheme for academia as a whole. Applied mathematics with an emphasis on theoretical tools is one domain where a lot of the negative aspects have been kept at bay. Unfortunately, it's also a field where statistically it is very hard to get a reasonably stable job. As far as areas of math go, I greatly enjoy theoretical computer science, probability theory, and continuous math that's useful for signal processing (complex analysis, Fourier series, functional analysis, machine learning, etc.)
I had not seen the previous post on career choice and will look into it. But the main reason for this thread was that I think that as far as getting a job and sustaining myself goes, I'm better off trying to hack my preferences and causing myself to actually enjoy computer programming, instead of finding it loathsome as I do now. This is based on a non-trivial amount of interaction with people in the start-up community, in academia, and at government research labs.
In one of the previous discussions, I suggested taking a job as a database/web developer at a university department. I think you don't actually need to hack yourself to enjoy computer programming to do this, because if you're a fast programmer you can finish your assignments in a small fraction of the time that's usually assigned, which leaves you plenty of time to do whatever else you want. So if you just want to get a job and sustain yourself, that seems like something you should consider.
But that advice doesn't take into account your interest in FAI and...
I was inspired by the recent post discussing self-hacking for the purpose of changing a relationship perspective to achieve a goal. Despite my feeling inspired, though, I also felt like life hacking was not something I could ever want to do even if I perceived benefits to doing it. It seems to me that the place where I would need to begin is hacking myself in order to cause myself to want to be hacked. But then I started contemplating whether this is a plausible thing to do.
In my own case, there are two concrete examples in mind. I am a graduate student working on applied math and probability theory in the field of machine vision. I was one of those bright-eyes, bushy-tailed dolts as an undergrad who just sort of floated to grad school believing that as long as I worked sufficiently hard, it was a logical conclusion that I would get a tenure-track faculty position at a desirable university. Even though I am a fellowship award winner and I am working with a well-known researcher at an Ivy League school, my experience in grad school (along with some noted articles) has forced me to re-examine a lot of my priorities. Tenure-track positions are just too difficult to achieve and achieving them is based on networking, politics, and whether the popularity of your research happens to have a peak at the same time that your productivity in that area also has a peak.
But the alternatives that I see are: join the consulting/business/startup world, become a programmer/analyst for a large software/IT/computer company, work for a government research lab. I worked for two years at MIT's Lincoln Laboratory as a radar analyst and signal processing algorithm developer prior to grad school. The main reason I left that job was because I (foolishly) thought that graduate school was where someone goes to specifically learn the higher-level knowledge and skills to do theoretical work that transcends the software development / data processing work that is so common. I'm more interested in creating tools that go into the toolbox of an engineer than with actually using those tools to create something that people want to pay for.
I have been deeply thinking about these issues for more than two years now, almost every day. I read everything that I can and I try to be as blunt and to-the-point about it as I can be. Future career prospects seem bleak to me. Everyone is getting crushed by data right now. I was just talking with my adviser recently about how so much of the mathematical framework for studying vision over the last 30 years is just being flushed down the tubes because of the massive amount of data processing and large scale machine learning we can now tractably perform. If you want to build a cup-detector for example, you can do lots of fancy modeling, stochastic texture mapping, active contour models, fancy differential geometry, occlusion modeling, etc. Or.. you can just train an SVM on 50,000,000 weakly labeled images of cups you find on the internet. And that SVM will utterly crush the performance of the expert system based on 30 years of research from amazing mathematicians. And this crushing effect only stands to get much much worse and at an increasing pace.
In light of this, it seems to me that I should be learning as much as I can about large-scale data processing, GPU computing, advanced parallel architectures, and the gross details of implementing bleeding edge machine learning. But, currently, this is exactly the sort of thing I hate and went to graduate school to avoid. I wanted to study Total Variation minimization, or PDE-driven diffusion models in image processing, etc. And these are things that are completely crushed by large data processing.
So anyway, long story short: suppose that I really like "math theory and teaching at a respected research university" but I see the coming data steamroller and believe that this preference will cause me to feel unhappy in the future when many other preferences I have (and some I don't yet know about) are effected negatively by pursuit of a phantom tenure-track position. But suppose also that another preference I have is that I really hate "writing computer code to build widgets for customers" which can include large scale data analyses, and thus I feel an aversion to even trying to *want* to hack myself and orient myself to a more practical career goal.
How does one hack one's self to change one's preferences when the preference in question is "I don't want to hack myself?"