Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Assessing Kurzweil: the results

42 Stuart_Armstrong 16 January 2013 04:51PM

Predictions of the future rely, to a much greater extent than in most fields, on the personal judgement of the expert making them. Just one problem - personal expert judgement generally sucks, especially when the experts don't receive immediate feedback on their hits and misses. Formal models perform better than experts, but when talking about unprecedented future events such as nanotechnology or AI, the choice of the model is also dependent on expert judgement.

Ray Kurzweil has a model of technological intelligence development where, broadly speaking, evolution, pre-computer technological development, post-computer technological development and future AIs all fit into the same exponential increase. When assessing the validity of that model, we could look at Kurzweil's credentials, and maybe compare them with those of his critics - but Kurzweil has given us something even better than credentials, and that's a track record. In various books, he's made predictions about what would happen in 2009, and we're now in a position to judge their accuracy. I haven't been satisfied by the various accuracy ratings I've found online, so I decided to do my own assessments.

I first selected ten of Kurzweil's predictions at random, and gave my own estimation of their accuracy. I found that five were to some extent true, four were to some extent false, and one was unclassifiable 

But of course, relying on a single assessor is unreliable, especially when some of the judgements are subjective. So I started a call for volunteers to get assessors. Meanwhile Malo Bourgon set up a separate assessment on Youtopia, harnessing the awesome power of altruists chasing after points.

The results are now in, and they are fascinating. They are...

continue reading »

Kurzweil's predictions: good accuracy, poor self-calibration

31 Stuart_Armstrong 11 July 2012 09:55AM

Predictions of the future rely, to a much greater extent than in most fields, on the personal judgement of the expert making them. Just one problem - personal expert judgement generally sucks, especially when the experts don't receive immediate feedback on their hits and misses. Formal models perform better than experts, but when talking about unprecedented future events such as nanotechnology or AI, the choice of the model is also dependent on expert judgement.

Ray Kurzweil has a model of technological intelligence development where, broadly speaking, evolution, pre-computer technological development, post-computer technological development and future AIs all fit into the same exponential increase. When assessing the validity of that model, we could look at Kurzweil's credentials, and maybe compare them with those of his critics - but Kurzweil has given us something even better than credentials, and that's a track record. In various books, he's made predictions about what would happen in 2009, and we're now in a position to judge their accuracy. I haven't been satisfied by the various accuracy ratings I've found online, so I decided to do my own.

Some have argued that we should penalise predictions that "lack originality" or were "anticipated by many sources". But hindsight bias means that we certainly judge many profoundly revolutionary past ideas as "unoriginal", simply because they are obvious today. And saying that other sources anticipated the ideas is worthless unless we can quantify how mainstream and believable those sources were. For these reasons, I'll focus only on the accuracy of the predictions, and make no judgement as to their ease or difficulty (unless they say things that were already true when the prediction was made).

Conversely, I won't be giving any credit for "near misses": this has the hindsight problem in the other direction, where we fit potentially ambiguous predictions to what we know happened. I'll be strict about the meaning of the prediction, as written. A prediction in a published book is a form of communication, so if Kurzweil actually meant something different to what was written, then the fault is entirely his for not spelling it out unambiguously.

One exception to that strictness: I'll be tolerant on the timeline, as I feel that a lot of the predictions were forced into a "ten years from 1999" format. So I'll estimate the prediction accurate if it happened at any point up to the end of 2011, if data is available. 

The number of predictions actually made seem to vary from source to source; I used my copy of "The Age of Spiritual Machines", which seems to be the original 1999 edition. In the chapter "2009", I counted 63 prediction paragraphs. I then chose ten numbers at random between 1 and 63, and analysed those ten predictions for correctness (those wanting to skip directly to the final score can scroll down). Seeing Kurzweil's nationality and location, I will assume all prediction refer only to technologically advanced nations, and specifically to the United States if there is any doubt. Please feel free to comment on my judgements below; we may be able to build a Less Wrong consensus verdict. It would be best if you tried to reach your own conclusions before reading my verdict or anyone else's. Hence I present the ten predictions, initially without commentary:

continue reading »

"Outside View!" as Conversation-Halter

49 Eliezer_Yudkowsky 24 February 2010 05:53AM

Followup toThe Outside View's Domain, Conversation Halters
Reply toReference class of the unclassreferenceable

In "conversation halters", I pointed out a number of arguments which are particularly pernicious, not just because of their inherent flaws, but because they attempt to chop off further debate - an "argument stops here!" traffic sign, with some implicit penalty (at least in the mind of the speaker) for trying to continue further.

This is not the right traffic signal to send, unless the state of knowledge is such as to make an actual halt a good idea.  Maybe if you've got a replicable, replicated series of experiments that squarely target the issue and settle it with strong significance and large effect sizes (or great power and null effects), you could say, "Now we know."  Or if the other is blatantly privileging the hypothesis - starting with something improbable, and offering no positive evidence to believe it - then it may be time to throw up hands and walk away.  (Privileging the hypothesis is the state people tend to be driven to, when they start with a bad idea and then witness the defeat of all the positive arguments they thought they had.)  Or you could simply run out of time, but then you just say, "I'm out of time", not "here the gathering of arguments should end."

But there's also another justification for ending argument-gathering that has recently seen some advocacy on Less Wrong.

An experimental group of subjects were asked to describe highly specific plans for their Christmas shopping:  Where, when, and how.  On average, this group expected to finish shopping more than a week before Christmas.  Another group was simply asked when they expected to finish their Christmas shopping, with an average response of 4 days.  Both groups finished an average of 3 days before Christmas.  Similarly, Japanese students who expected to finish their essays 10 days before deadline, actually finished 1 day before deadline; and when asked when they had previously completed similar tasks, replied, "1 day before deadline."  (See this post.)

Those and similar experiments seem to show us a class of cases where you can do better by asking a certain specific question and then halting:  Namely, the students could have produced better estimates by asking themselves "When did I finish last time?" and then ceasing to consider further arguments, without trying to take into account the specifics of where, when, and how they expected to do better than last time.

From this we learn, allegedly, that "the 'outside view' is better than the 'inside view'"; from which it follows that when you're faced with a difficult problem, you should find a reference class of similar cases, use that as your estimate, and deliberately not take into account any arguments about specifics.  But this generalization, I fear, is somewhat more questionable...

continue reading »

View more: Next