"How We're Predicting AI — or Failing to"

lukeprog

The new paper by Stuart Armstrong (FHI) and Kaj Sotala (SI) has now been published (PDF) as part of the Beyond AI conference proceedings. Some of these results were previously discussed here. The original predictions data are available here.

Abstract:

This paper will look at the various predictions that have been made about AI and propose decomposition schemas for analysing them. It will propose a variety of theoretical tools for analysing, judging and improving these predictions. Focusing speciﬁcally on timeline predictions (dates given by which we should expect the creation of AI), it will show that there are strong theoretical grounds to expect predictions to be quite poor in this area. Using a database of 95 AI timeline predictions, it will show that these expectations are born out in practice: expert predictions contradict each other considerably, and are indistinguishable from non-expert predictions and past failed predictions. Predictions that AI lie 15 to 25 years in the future are the most common, from experts and non-experts alike.

Abstract:

This paper will look at the various predictions that have been made about AI and propose decomposition schemas for analysing them. It will propose a variety of theoretical tools for analysing, judging and improving these predictions. Focusing speciﬁcally on timeline predictions (dates given by which we should expect the creation of AI), it will show that there are strong theoretical grounds to expect predictions to be quite poor in this area. Using a database of 95 AI timeline predictions, it will show that these expectations are born out in practice: expert predictions contradict each other considerably, and are indistinguishable from non-expert predictions and past failed predictions. Predictions that AI lie 15 to 25 years in the future are the most common, from experts and non-experts alike.

The original data can be found via: http://lesswrong.com/lw/e79/ai_timeline_prediction_data/

(much better to use that than to squint at the pictures!)

My subjective impressions: predictors very rarely quote or reference each other when making predictions. Many predictions seem purely an individual guess. I've seen no sign of an expert consensus, or of much experts critiquing or commending each other's work. I really feel that predicting AI has not been seen as something where anyone should listen to other people's opinions. There are some exceptions - Kurzweil, for instance, seems famous enough that people are willing to quote his estimates, usually to claim he got it wrong - but too few.

I'm actually thinking that the "non-experts were no better than experts" bit is maybe a little misleading, as I remember seeing a lot of the non-experts base their predictions on what experts had been saying.

3gwern14y

They may not cite each other, but the influence can still be there as background reading etc. I may not cite Legge when I think there's a good chance of breakthroughs in the 2020s but the influence is there (well, it was until I mentioned him just now). To give a real-world example, compiling http://www.gwern.net/2012%20election%20predictions I know that the forecasters were all reading each others' blogs or twitters etc because in scouring their sites I see enough cross-links or similar topics, but anyone who looked at just the relevant pages of predictions or prediction CSVs would miss that completely and think they were deriving their similar predictions from independent models. I think there's a lot of shared ideas and reading which rarely is explicitly cited in the same passage as a specific prediction with the exception of really offensive estimates like Kurzweil's self-promoting (have you been reading the reviews of his latest book? Everyone's dragging out Hofstadter's old dog shit quote, which one can't help but feel that he would not have been so explicit and crude if Kurzweil didn't really rub him the wrong way). But I don't know how one would test the consensus idea other than waiting and seeing whether expert predictions continue to cluster around 2040 even as we hit 2020s and 2030s.

19

"How We're Predicting AI — or Failing to"

19

19

19

"How We're Predicting AI — or Failing to"

19

19