Over the past few months, the Singularity Institute has published many papers on topics related to Friendly AI. It's wonderful that these ideas are getting written up, and it's virtually always better to do something suboptimal than to do nothing. However, I will make the case below that academic papers are a terrible way to discuss Friendly AI, and other ideas in that region of thought space. We need something better.
I won't try to argue that papers aren't worth publishing. There are many reasons to publish papers - prestige in certain communities and promises to grant agencies, for instance - and I haven't looked at them all in detail. However, I think there is a conclusive case that as a discussion forum - a way for ideas to be read by other people, evaluated, spread, criticized, and built on - academic papers fail. Why?
1. The time lag is huge; it's measured in months, or even years.
Ideas structured like the Less Wrong Sequences, with large inferential distances between beginning and ending, have huge webs of interdependencies: to read A you have to read B, which means you need to read C, which requires D and E, and on and on and on. Ideas build on each other. Einstein built on Maxwell, who built on Faraday, who built on Newton, who built on Kepler, who built on Galileo and Copernicus.
For this to happen, ideas need to get out there - whether orally or in writing - so others can build on them. The publication cycle for ideas is like the release cycle for software. It determines how quickly you can get feedback, fix mistakes, and then use whatever you've already built to help make the next thing. Most academic papers take months to write up, and then once written up, take more months to publish. Compare that to Less Wrong articles or blog posts, where you can write an essay, get comments within a few hours, and then write up a reply or follow-up the next day.
Of course, some of that extra time lag is that big formal documents are sometimes needed for discussion, and big formal documents take a while. But academic papers aren't just limited by writing and reviewing time - they still fundamentally operate on the schedule of the seventeenth-century Transactions of the Royal Society. When Holden published his critique of the Singularity Institute on Less Wrong, a big formal document, Eliezer could reply with another big formal document in about three weeks.
2. Most academic publications are inaccessible outside universities.
This problem is familiar to anyone who's done research outside a university. The ubiquitous journal paywall. People complain about how the New York Times and Wall Street Journal have paywalls, but at least you can pay for them if you really want to. It isn't practical for almost anyone doing research to pay for the articles they need out-of-pocket, since journals commonly charge $30 or more per article, and any serious research project involves dozens or even hundreds of articles. Sure, there are ways to get around the system, and you can try to publish (and get everyone else in your field to publish) in open-access journals, but why introduce a trivial inconvenience?
3. Virtually no one reads most academic publications.
This obviously goes together with point #2, but even within universities, it's rare for papers, dissertations or even books to be read outside a very narrow community. Most people don't regularly read journals outside their field, let alone outside their department. Academic papers are hard to get statistics on, but eg., I was a math major in undergrad, and I can't even understand the titles of most new math papers. More broadly, the print run of most academic books is very small, only a few hundred or so. The average Less Wrong post gets more views than that.
4. It's very unusual to make successful philosophical arguments in paper form.
When doing research for Personalized Medicine, I often read papers to discover the results of some experiment. Someone gave drug X to people with disease Y. What were the results? How many were cured? How many had side effects? What were the costs and benefits? All useful information.
However, most recent Singularity Institute papers are neither empirical ("we did experiment X, these are the results") or mathematical ("if you assume A, B, and C, then D and E follow"). Rather, they are philosophical, like Paul Graham's essays. I honestly can't think of a single instance where I was convinced of an informal, philosophical argument through an academic paper. Books, magazines, blog posts - sure, but papers just don't seem to be a thing.
5. Papers don't have prestige outside a narrow subset of society.
Several other arguments here - the time lag, for instance - also apply to books. However, society in general recognizes that writing a book is a noteworthy achievement, especially if it sells well. A successful author, even if not compensated well, is treated a little like a celebrity: media interviews, fan clubs, crazy people writing him letters in green ink, etc. (This is probably related to them not being paid well: in the labor market, payment in social status probably substitutes to a high degree for payment in money, as we see with actors and musicians.)
There's nothing comparable for academic papers. No one ever writes a really successful paper, and then goes on The Daily Show, or gets written up in the New York Times, or gets harassed by crowds of screaming fangirls. (There are a few exceptions, like medicine, but philosophy and computer science are not among them.) Eg., a lot of people are familiar with Ioannidis's paper, Why most published research findings are false. However, he also wrote another paper, a few years earlier, titled Replication validity of genetic association studies. This paper actually has more citations - over 1300 at least count. But not only have we not heard of it, no one else outside the field has either. (Try Googling it, and you'll see what I mean.)
6. Getting people to read papers is difficult.
Most intellectual people regularly read books, blogs, newspapers, magazines, and other common forms of memetic transmission. However, it's much less common for people to read papers, and that reduces the affordances that people have for doing so, if they are asking "hey, this thing is a crazy idea, why should I believe it?". Papers are, intentionally, written for an audience of specialists rather than a general interest group, which reduces both the tendency and ability of non-specialists to read them when asked (and also violates the "Explainers shoot high - aim low" rule).
7. Academia selects for conformity.
The whole point of tenure is to avoid selecting for conformity - if you have tenure, the theory goes, you can work on whatever you want, without fear of being fired or otherwise punished. However, only a small (and shrinking) number of academics have tenure. In order to make sure fools didn't get tenure, it turns out academia resorted to lots and lots of negative selection. The famous letter by chemistry professor Erick Carreira illustrates some of what the selection pressure is like, similar to medicine or investment banking: there's a single, narrow "track", and people who deviate at any point are pruned. Lee Smolin has written about this phenomenon in string theory, in his famous book The Trouble with Physics.
Things may change in the future, but as it stands now, many ideas like the Singularity are non-conformist, well outside the mainstream. They aren't likely to go very far in an environment where deviations from the norm are seen negatively.
8. The current community isn't academic in origin.
This isn't an airtight argument, because it's heuristic - "things which worked well before will probably work again". However, heuristic arguments still have a lot of validity. One of the key purposes of a discussion forum, like Less Wrong or the SL4 list that was, is to get new people with bright ideas interested in the topics under discussion. Academia's track record of getting new people interested isn't that great - of the current Singularity Institute directors and staff, only one (Anna Salamon) has an academic background, and she dropped out of her PhD program to work for SIAI. What has been successful, so far, at bringing new people into our community? I haven't analyzed it in depth, but whatever the answer is, the priors are that it will work well again.
9. Our ideas aren't academic in origin.
Similarly to #8, this is a "heuristic argument" rather than an airtight proof. But I still think it's important to note that our current ideas about Friendly AI - any given AI will probably destroy the world, mathematical proof is needed to prevent that, human value is complicated and hard to get right, and so on - were not developed through papers, but through in-person and mailing list discussions (primarily). I'm also not aware of any ideas which came into our community through papers. Even science fiction has a better track record - eg. some of our key concepts originated in Vinge's True Names and Other Dangers. What formats have previously worked well for discussing ideas?
10. Papers have a tradition of violating the bottom line rule.
In a classic paper, one starts with the conclusion in the abstract, and then builds up an argument for it in the paper itself. Paul Graham has a fascinating essay on this form of writing, and how it came to be - it ultimately derives from the legal tradition, where one takes a position (guilty or innocent), and then defends it. However, this style of writing violates the bottom line rule. Once something is written on the paper, it is already either right or wrong, no matter what clever arguments you come up with in support of it. This doesn't make it wrong, of course, but it does tend to create a fitness environment where truth isn't selected for, just as Alabama creates a fitness environment where startups aren't selected for.
11. Academic moderation is both very strict and badly run.
All forums need some sort of moderation to avoid degenerating. However, academic moderation is very strict by normal standards - in a lot of journals, only a small fraction of submissions get approved. In addition, academic moderation has a large random element, and is just not very good overall; many quality papers get rejected, and many obvious errors slip through.
As if that wasn't enough, most journals are single-blind rather than double-blind. You don't know who the moderators are, but they know who you are, raising the potential for all kinds of obvious unfairness. The most common kind of bias is one that hurts us unusually badly: people from prestigious universities are given a huge leg up, compared to people outside the system.
(This article has been cross-posted to my blog, The Rationalist Conspiracy.)
EDIT #1: As Lukeprog notes in the comments, academic papers are not our main discussion forum for FAI ideas. In practice, the main forum is still in-person conversations. However, in-person conversations have critical limitations too, albeit more obvious ones. Some crucial limits are the small number of people who can participate at any one time; the lack of any external record that can be looked up later; the lack of any way to "broadcast" key findings to a larger audience (you can shout, but that's not terribly effective); and the lack of lots of time to think, since each participant in the conversation can't really wait three hours before replying.
EDIT #2: To give a specific example of an alternative forum for FAI discussion, I think the proposal for an AI Risk wiki would solve most of the problems listed here.
Great post - I definitely agree with some of your points. I'm very new to LW and haven't even written an introductory post yet, but I'm very impressed with what I've seen overall. I am even flying out to San Francisco tomorrow to discuss joining the newly-renamed CFAR. My background is entirely academic, as I have a PhD in experimental psychology and I'm interested in formalising some of the rationality measures CFAR is looking at. I even had a brief email exchange with Anna Salamon about the usefulness and validity of academic publications.
Here's my take on your points.
Yes, the time lag is huge. It's worth noting that in academia papers aren't used as discussion forums to share ideas rapidly - you read them to see what other labs are doing, and discuss them in journal clubs where graduate students and postdocs can learn about the field and get ideas for their own work. Conference talks and poster presentations are the usual format for discussions between academics from different labs. The academic community as a whole has been very slow to harness the power of the web for the formal discussion of research, although much does take place on science blogs, and now some of the journals are very belatedly starting to host their own discussion forums.
Yes. This is hugely irritating, most academics hate it, and it's the result of a publishing model that's massively outdated. Many governments are now taking steps to ensure that publicly-funded research is accessible to all, though we are by no means there yet.
Yes, papers aren't widely read outside the field they're published in. But they're actually not usually supposed to be - the work and the jargon in most fields is so specialised that just being able to read it and understand what people are talking about takes years of training. At that level you need a specialised vocabulary just to be able to ask the questions. Compare it to the jargon used here at LW - it took me a while to get into it, and most of it isn't completely inaccessible, but it's still tough for a first-timer to break into this community if they don't know much about rationalism.
Not sure about this. Philosophers make philosophical arguments in paper form all the time, and they do it primarily to convince other philosophers. If they want to convince people without the formal training, that's when they'll usually write a book. (As an experimentalist myself, and given that I'm interested primarily in testing rationality measures, it doesn't really apply.)
Yes and no to this one. It's true that writing a paper doesn't get you mobbed by hordes of screaming fangirls, as I know to my detriment! But again it's about who you're trying to convince. You need publications to be taken seriously by other academics, if that's what you want to do. And if you want to be taken seriously by wider society, there are certainly other ways to do that than by becoming an academic. But as I mentioned before, there's a vibrant community on science blogs and people interested in science who consider a paper to be the gold standard of scholarship.
Yes, actually you're supporting the point here I made in my reply to 3. The two tie together quite well in fact.
This is definitely a problem with academia, although I'd consider it more a feature than a bug. Basically the current system is optimised to stop crazy people coming in and messing things up. That is, if you have an idea that's novel but mainstream you'll attract funding; if you have an idea that's way out there you often won't, people will think you're a crackpot, etc. etc. GIven the existence of actual crackpots, this is a necessary defence mechanism, although perhaps some knowledge of the base rate of crackpots would be helpful so that academia could make decently Bayesian decisions...
This is true for the moment but as I've said I'm interested in joining CFAR (probably not SIAI though, it just doesn't interest me as much as rationality training) and I'm probably not the only academic who's at least a bit interested. The more research you can pump out the more either organisation will be seen as a viable place for academics to go and work. I'm sure we can bring things to the table that most non-academics cannot, and can learn things ourselves as well.
Disagree strongly. I suspect you're right with FAI, which is a topic I'm not very familiar with, but other topics about cognitive biases and heuristics, Bayesian decision theory and so on, have been mainstream in academia for some time now. In fact part of my PhD was based on the premise that the brain treats incoming information in a Bayesian way (see also Tom Griffiths' work at Berkeley: http://cocosci.berkeley.edu/tom/). So I don't think it's correct to say that all of the ideas in this community aren't academic in origin; it might be better to say that some of them aren't.
Yes and no. Of course in the abstract you have a conclusion - because the abstract is a tiny mini-paper that you can read to find the answer without having to plough through the whole thing. If you want to check the work, you go and do just that. Papers should be stories - they should flow well and communicate the idea they're trying to sell. But that idea isn't written down before the research is done, otherwise there'd be no point in doing the research. It might be written down in the paper before you reach the evidence that supports it, but that's for narrative reasons. I should note that many papers do start off with a question: "is it A or B? let's find out!" rather than stating outright whether A or B has won the day. But in reality one of them has already, because the universe is as it is, and the experiments you do to test it only reflect that.
Again this has its good and bad points. A fraction of submissions getting approval goes back to the 'crackpot filter' I talked about in my response to 7, which also explains why universities are privileged over other organisations. You don't want journals filled with any old rubbish, though some journals are obviously better than others (and in your field it very quickly becomes apparent which ones those are). So how do you keep quality up without filtering? You're right that good papers get rejected and errors slip through - but generally that stuff is caught pretty fast and corrected, and eventually a whole field changes when the errors are brought to its attention. A good example is neuroimaging using fMRI, which a decade ago you could publish anything in really easily but now your stats have to be pretty much watertight. And you're right that it's unfair that reviewers get to know who you are but you don't get to know who they are for the most part... means they can block your grants or your papers if they don't 'like your ideas or think you're getting too close to their turf. We are, after all, hierarchical social apes, even those of us who try to be rational.
To summarise then, I generally agree for the most part but there are nuances beyond what you've stated in your arguments. I'd agree that papers aren't great for discussion among the community at the time they're published, but the advantage is that anyone can go back and find them and other papers and build a narrative from that, much more easily (at the moment) than can be done with a blog. The signal-to-noise ratio is generally higher, although a blog like this with good moderation and smart, curious people ameliorates many problems.