First, I must apologize for the somewhat self-serving post, but as it is in the discussion section I hope that this can be forgiven.
I think that its entirely appropriate for rationalists to ask each other for help.
One thing I didn't see you mention is neuroscience. My understanding is that some AGI researchers are currently taking this route; e.g. Shane Legg, mentioned in another comment, is an AGI researcher who is currently studying theoretical neuroscience with Peter Dayan. Demis Hassabis is another person interested in AGI who's taking the neuroscience route (see his talk on this subject from the most recent Singularity Summit). I'm personally interested in FAI, and I suspect that we need to study the brain to understand in more detail the nature of human preference. In terms of a career path, it's possible I'll go to graduate school at some point in the future, but my current plans are to just get a programming job and study neuroscience in my free time.
Have you given a thought to just taking the day job route? There are some problems, as I've found more than a few journal articles locked behind a paywall, but there are some ways for dealing with this. Furthermore, I've found a surprising number of recent neuro articles are available through open access journals like PNAS, Frontiers and through other routes (Google, Google Scholar, CiteseerX, author websites). If you're interested more in CS research, then I suspect you'll have even less trouble; for some reason recent (CS papers) seem to almost always be available over the internet.
As far as neuroscience goes, yes I have strongly considered it. I think that I would like to do a program in computational neuroscience. The joint program at U Pitt and Carnegie Mellon looks interesting for this sort of thing, of course MIT and Caltech both have solid programs but I am not confident that my record is strong enough to get into either of those schools.
The day job route makes me somewhat nervous because: (a) I'm not sure how difficult it is to get published without the right background/support (b) I'm worried that I'll be isolated from other researchers who might have insight I could benefit from
What do you plan to do in AGI? The problem with AGI is that many of the fundamental tools necessary for AGI are only in a preliminary state of development. If you're going to end up working on developing those tools, you might as well do so in a more lucrative area, such as statistics.
you might as well do so in a more lucrative area, such as statistics.
I was going to launch into some of my interests, but you've got my attention. Could you expand on this point more? I have an idea or two about why you might be saying that statistics is more lucrative, but I want to know what your point off view is here.
Money is of course only one factor; I think you could easily make a career in statistics without compromising your interests.
There are many researchers in statistics who work on areas which are extremely relevant to artificial intelligence. One of the best examples is Cosma Shalizi (http://www.cscs.umich.edu/~crshalizi/), who developed the "Causal State Splitting Reconstruction" algorithm for automatically detecting patterns in time series.
Your background is also quite suited for statistics: both the computer science and the philosophy. Nowadays, the computational element is becoming more and more prominent in statistics; the best example is the rapid growth of Markov Chain Monte Carlo sampling for Bayesian analysis. MCMC methods originated in statistical physics, but they are now being employed to analyze complex probability models in exotic settings such as social networks.
As for the philosophical aspect of statistics (which is usually understated,) you would be surprised at how many foundational questions still remain on the question of how to draw conclusions from data. There are a few basic tenets for statistical procedures, e.g. the likelihood principle and the conditionality principle, but none of them are universally accepted as "axioms" and in fact some of them contradict each other. The choice of which principle to adopt results in an explosion of competing methodologies: frequentist, Bayesian, empirical Bayesian, conditional, "pragmatic" (cross-validation) etc. Yet the question of how to do inference from data is obviously a central question to be addressed for the goal of designing an inductive agent.
That is interesting. I was aware of the growth of Monte Carlo sampling due to its use in Go AI, and I have an interest in the philosophical aspects of statistical inference though I haven't follwed up on it quite to the extent that I would like. One of the things I'm currently picking at is An Introduction to Kolmogorov Complexity by Li and Vitanyi. I started looking into it after seeing the discussions about complexity and Solomonoff induction on this site.
However, I am still unclear as to why this is more lucrative than other areas related to AGI.
Ah, now I see what angle you are coming at this from. Yes, data mining techniques would certainly be invaluable to an AGI (which needs to be able to organize its input data into useful information) as well as lucrative to any number of companies. I saw a talk by John Hopcroft a while back that made it seem very appealing.
It may not be quite what you asked for, but I still want to direct you to an interesting read.
If you want get paid to work on the less-computer-sciencey layers of AI you probably do need to be in academia, but if you are fine with the more-computer-sciencey layers (maybe collaborating on the others in spare time) you can probably work at any ordinary (as opposed to general) AI company, like Google if you can manage it.
Well, I appreciate it, but I've read a good portion of that already (still working through it, since I've got a pretty wide reading list and a lot of free time right now). I am more interested in working at a theoretical level to some degree, though implementation is interesting to me as well of course.
I do not have much interest in working on ordinary AI stuff, I would rather go into Cognitive Science or maybe even applied mathematics. I have a reasonably broad knowledge of theory already (graduate courses in mathematical logic, theory of computation and complexity theory, several independent studies on algebra and algebraic geometry, working on categorical logic and statistical inference/decision theory right now), and my inclination is more at the level of theory than making fun or useful ordinary AI applications.
Plus, I'm pretty sure that in order to get a really great job at Google I would need my PhD anyway (correct me if I'm wrong, that's just the sort of impression I got when looking at the options there) and there is no way that my coding skills up to par with what Google expects.
I work at Google and don't have a PhD :-) But if you're more interested in basic theory, IBM Research or Microsoft Research would be better choices, IMO. Or just go into academia.
computer science programs with strong AGI groups
I'm not under the impression there are a lot of those. You might want to check out the Gatsby unit, maybe follow the breadcrumb trail starting with Shane Legg.
I appreciate the link! I've seen his name come up before, but never looked further. Maybe I'll be able to pull out some useful, relevant information that I can share here.
Hi Zetetic! Your curriculum is very interesting, in particular referring to "the link between category theory and cognitive science". Are you talking about this? Or something else? I'm quite fond of category theory but AFAICR I've never stumbled upon such a link... Any pointer is really appreciated.
Here is something I found interesting. The author has a number of papers on this sort of thing. This one is pretty interesting as well. This site in particular led me to a great number of links.
This seemed to be of interest on the more neuroscience-y side of things (though it seems like it is very much along the same lines as the material above). So did this.
I haven't had time to review all (or even most) of that material, so I don't know what is good and what is bad. What interests me is that it appears that a lot of researchers like the idea, so it seems reasonable to assume that there is a good chance they are on to something. So, I've been spending a portion of my time researching this and studying category theory.
Man that is super interesting.
This almost wants me to switch from arithmetic geometry to category theory...
Well, if you've already got the algebraic background needed for arithmetic geometry then you could probably break into the basics fairly painlessly. I found that my familiarity with algebraic structures from the algebraic geometry side of things has made my readings much smoother than they were the first time I tried to tackle it (near the end of my sophomore year I think?), which was very slow going (I was using Maclane's CWM and simply didn't have the appropriate background) and I did not get much out of it. Even if you're already doing your dissertation or something, it hardly seems like too much of a stretch to switch over.
I'm much earlier than that (only my first year in grad school), it definitely wouldn't be a problem in terms of my mathematical development to switch to category theory, but there are no real pure category theorists at my school and I've just committed to working with a somewhat famous number theorist who has promised to contribute significant money to me gallivanting about Europe this summer at conferences.
I have similar backgorund (although probably less far along with it) and certainly would find working in AGI to be by far the most meaningful and motivating thing possible, but I'm somewhat more sceptical that I'm the very best candidate to do something so unimaginable important and sensitive. I mean, it's a **ing FAI, not a silly little nuke or genome or other toy like that.
I'm somewhat more sceptical that I'm the very best candidate to do something so unimaginable important and sensitive. I mean, it's a **ing FAI, not a silly little nuke or genome or other toy like that.
So work on FAI but don't actually run one. Get people who you think are more qualified to check your work if you make useful progress.
The problem appears to be that no one has a clue how to work on FAI and make sure that it actually is FAI. If someone made what they thought was FAI and it wasn't actually FAI, how could you tell until it was too late?
If someone made what they thought was FAI and it wasn't actually FAI, how could you tell until it was too late?
How can you tell that a theorem is correct? By having a convincing proof, or perhaps other arguments to the effect of the theorem being correct, and being trained to accept correct proofs (arguments) and not incorrect ones.
Even though we currently have no idea about how to construct a FAI or how an argument for a particular design being a FAI might look like, we can still refuse to believe that something is a FAI when we are not given sufficient grounds for believing it is one, provided the people who make such decisions are skilled enough to accept correct arguments but not incorrect ones.
The rule to follow is to not actually build or run things that you don't know to be good. By default, they don't even work, and if they do, they are not good, because value is complex and won't be captured by chance. There need to be strong grounds for believing that a particular design will work, and here rationality training is essential, because by default people follow one of the many kinds of crazy reasoning about these things.
If someone made what they thought was FAI and it wasn't actually FAI, how could you tell until it was too late?
See! You've found a problem to work on already! :)
[The downvote on your comment isn't mine btw.]
I read a science fiction story where they made a self sustaining space station. Placed a colony of scientist and engineers needed to run it and then sealed it off with no connection to the outside world. Then they modified all the computer files to make it appear as though the humans had evolved on the station and there was nothing else but the station. Then they woke up the AI and start stress testing it by attacking it in non-harmful ways.
It was an interesting story, not sure how useful it would be in real life. The AI actually manages to figure out that there is stuff outside the station and they are only saved because it creates its own moral code in which killing is wrong. This was a very convenient plot point so I wouldn't trust it in real life.
How do you know that a person is really friendly? You use methods that have worked in the past and look for manipulative techniques that misleadingly friendly people use to make you think they are friendly. We know that someone is friendly via the same methodology that we determine what it means to be friendly, subjective benefits (emotional support etc.) and goal assistance (helping you move when they could simply refuse to do so) without malicious motives that ultimately disservice you.
In the case of FAI we want more surety, and we can presumably get this via simulation and proofs of correctness. I would assume that even after we had a proof of correctness for a meta-ethical system we would want to run it through as many virtual scenarios as possible, since the human brain is simply not capable of the chains of reasoning within the meta-ethics that the machine would be, so we would want to introduce it to scenarios that are as complex as possible in order to determine that it fits our intuition of friendliness.
It seems to me that the bulk of the work is in the arena of identifying the most friendly meta-ethical architecture. The Lokhorst paper lukeprog posted a while ago clarified a few things for me, though I have no access to the cutting edge work on FAI (save for what leaks out into the blog posts), and judging by what Will Newsome has said in the past (cannot find the post) they have compiled a relatively large list of possibly relevant sub-problems that I would be very interested to see (even if many of them are likely to be time drains).
Hmm, it seems like other researchers could at least assess you. Besides, it isn't like there aren't always sub-problems for those who feel less capable. You don't have to be the head researcher to do useful things related to a large project, you can have someone who has established themselves direct you until you get a sense for what you are capable of.
Thanks,and yea that MIGHT work... But so far everything Eliezer have said indicates the opposite and argues it very well.
I'm mostly hoping for some VERY far removed sub-sub-problem maybe.
Hmm. I've read most of the sequences and some of his monograph on FAI, but I don't recall him explicitly arguing against dividing up the work into sub-problems. Intuitively it seems that if you trust person X to cautiously do FAI then you should trust them to be able to pick out sub-problems and be able to determine when it is the case that they have been solved satisfactorily.
Could you point me out to the relevant links?
Also, I might be terribly mistaken here, but it seems like not every component of the AGI puzzle need be tied directly to FAI, at least in the development phase. Each one must be fully understood and integrated into an FAI, but I don't see why, say, I need to be as careful when designing a concept model of an ontology engine that can combine and generalize concepts, at least until I want to integrate it into the FAI, at which point the FAI people could review the best attempts at the various components of AGI and figure out whether any of them are acceptable and then how to integrate them into an FAI framework.
I guess so, but I distinctly remember some writing Eleizer did what gave the strong impression that if your IQ was below 9000 you shouldn't try to do anything but give the SIAI money. I don't remember from where thou and it certainly sounds weird, so maybe my memory just messed up.
Going solely on what Eliezer has said about 'exceeding your role models', I would take that with a grain of salt. I've never met Eliezer, but although he comes off as extremely intelligent, judging by his writings and level of achievement (which are impressive) he still does not come off to me as, say, Von Neumann intelligent.
Eliezer's writings have clarified my thoughts a great deal and given me a stronger sense of purpose. He is a very intelligent researcher and gifted explainer and evangelist, but I don't take his word as Gospel, I take it as generally very good advice.
I've never met Eliezer, but although he comes off as extremely intelligent, judging by his writings and level of achievement (which are impressive) he still does not come off to me as, say, Von Neumann intelligent.
Wouldn't dispute that.
I've never met Eliezer, but although he comes off as extremely intelligent, judging by his writings and level of achievement (which are impressive) he still does not come off to me as, say, Von Neumann intelligent.
Wouldn't dispute that.
I seem to recall you saying as much - at one time as not quite like a thousand year old vampire and at another as not 'glittery'. It only occurs to me now that that combination make Jaynes a thousand year old Twilight vampire. Somehow that takes some of the impressiveness out of the metaphor, Luminosity revamp (cough) or not!
I suspect but am not certain that you're thinking of "So You Want To Be A Seed AI Programmer". I also suspect that the document is at least partially out of date, however.
Follow up to: Best career models for doing research?
First, I must apologize for the somewhat self-serving post, but as it is in the discussion section I hope that this can be forgiven. Also, I would not be surprised if there are at least a few college age people lurking around with very similar problems/issues, so I expect that this might prove very useful to at least a couple of people here. If this works out, I do hope to eventually put it into the form of a more general top-level post on career advice for those interested in a career in AGI.
Now, on to the issue:
It has come to my attention that research opportunities in AGI appear to both be somewhat limited, and somewhat unstructured compared to more well-developed fields that I have looked into. It seems to me that it would be useful to have a discussion here, given the unusual population density of AGI enthusiasts/professionals, about the possible pathways that one might take after the completion of an undergraduate degree. In my case, I have a strong background in mathematics, computer science and philosophy as well as a growing knowledge base in psychology. I've been studying Pearl's work, Timeless Decision Theory, cognitive science, evolutionary and cognitive psychology, Bishop's book on Pattern Recognition and Machine Learning, the link between category theory and cognitive science/AI (which appears to have some promise for building ontologies that can combine concepts and generalize), game theory, probability/statistics, computational complexity and I have been trying to get a few more programming languages under my belt.
My initial impulse was to go ahead and study for, and then take, all of the relevant GRE subject tests (Mathematics, Psychology and Computer science anyway) and apply to cognitive science and computer science programs with strong AGI groups. I've found that the latter option is more difficult that I had realized, which is somewhat disheartening, as my future planning model does not seem to work in such an underdeveloped field, and there is no easy to find established standard source for finding out which schools/programs to look at. I also realized that the former option does not necessarily conform to my research interestes as much as I would like it to, this being a fairly long term commitment.
Perhaps I lack the knowledge to successfully evaluate AGI programs; perhaps in the case of this particular area getting a PhD is not the best option; perhaps if I were more knowledgeable or wiser I might be better able to navigate where to go next, but I seem to be at a loss here. So; I come to you, fellow Less Wrongians, in search of guidance. Can any of you help to point me (and hopefully plenty of others) in the right (or at least less wrong) direction?