Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Assessing Kurzweil: the results

42 Stuart_Armstrong 16 January 2013 04:51PM

Predictions of the future rely, to a much greater extent than in most fields, on the personal judgement of the expert making them. Just one problem - personal expert judgement generally sucks, especially when the experts don't receive immediate feedback on their hits and misses. Formal models perform better than experts, but when talking about unprecedented future events such as nanotechnology or AI, the choice of the model is also dependent on expert judgement.

Ray Kurzweil has a model of technological intelligence development where, broadly speaking, evolution, pre-computer technological development, post-computer technological development and future AIs all fit into the same exponential increase. When assessing the validity of that model, we could look at Kurzweil's credentials, and maybe compare them with those of his critics - but Kurzweil has given us something even better than credentials, and that's a track record. In various books, he's made predictions about what would happen in 2009, and we're now in a position to judge their accuracy. I haven't been satisfied by the various accuracy ratings I've found online, so I decided to do my own assessments.

I first selected ten of Kurzweil's predictions at random, and gave my own estimation of their accuracy. I found that five were to some extent true, four were to some extent false, and one was unclassifiable 

But of course, relying on a single assessor is unreliable, especially when some of the judgements are subjective. So I started a call for volunteers to get assessors. Meanwhile Malo Bourgon set up a separate assessment on Youtopia, harnessing the awesome power of altruists chasing after points.

The results are now in, and they are fascinating. They are...

continue reading »

Checking Kurzweil's track record

12 Stuart_Armstrong 30 October 2012 11:07AM

Predictions are cheap and easy; verification is hard, essential, and rare. For things like AI, we seem to be restricted to nothing but expert predictions - but expert predictions on AI are not very good, either in theory or in practice. If we are some experts who stand out, we would really want to identify them - and there is nothing better than a track record for identifying true experts.

So we're asking for help to verify the predictions of one of the most prominent futurists of this century: Ray Kurzweil, from his book "The Age of Spiritual Machines". By examining his predictions for times that have already come and gone, we'll be able to more appropriately weight his predictions for times still to come. By taking part, by lending your time to this, you will be directly helping us understand and predict the future, and will get showered in gratitude and kudos and maybe even karma.

I've already made an attempt at this (if you are interested in taking part in this project, avoid clicking on that link for now!). But you cannot trust a single person's opinions, and that was from a small (albeit random) sample of the predictions. For this project, I've transcribed his predictions into 172 separate (short) statements, and any volunteers would be presented with a random selection among these. The volunteers would then do some Google research (or other) to establish whether the prediction had come to pass, and then indicate their verdict. More details on what exactly will be measured, and how to interpret ambiguous statements, will be given to the volunteers once the project starts.

If you are interested, please let me know at stuart.armstrong@philosophy.ox.ac.uk (or in the comment thread here), indicating how many of the 172 questions you would like to attempt. The exercise will probably happen in late November or early December.

This will be done unblinded, because Kurzweil's predictions are so well known that it would be infeasible to find large numbers of people who are technologically aware but ignorant of them. Please avoid sharing your verdicts with others; it is entirely your own individual assessment that we are interested in having.

Kurzweil's predictions: good accuracy, poor self-calibration

30 Stuart_Armstrong 11 July 2012 09:55AM

Predictions of the future rely, to a much greater extent than in most fields, on the personal judgement of the expert making them. Just one problem - personal expert judgement generally sucks, especially when the experts don't receive immediate feedback on their hits and misses. Formal models perform better than experts, but when talking about unprecedented future events such as nanotechnology or AI, the choice of the model is also dependent on expert judgement.

Ray Kurzweil has a model of technological intelligence development where, broadly speaking, evolution, pre-computer technological development, post-computer technological development and future AIs all fit into the same exponential increase. When assessing the validity of that model, we could look at Kurzweil's credentials, and maybe compare them with those of his critics - but Kurzweil has given us something even better than credentials, and that's a track record. In various books, he's made predictions about what would happen in 2009, and we're now in a position to judge their accuracy. I haven't been satisfied by the various accuracy ratings I've found online, so I decided to do my own.

Some have argued that we should penalise predictions that "lack originality" or were "anticipated by many sources". But hindsight bias means that we certainly judge many profoundly revolutionary past ideas as "unoriginal", simply because they are obvious today. And saying that other sources anticipated the ideas is worthless unless we can quantify how mainstream and believable those sources were. For these reasons, I'll focus only on the accuracy of the predictions, and make no judgement as to their ease or difficulty (unless they say things that were already true when the prediction was made).

Conversely, I won't be giving any credit for "near misses": this has the hindsight problem in the other direction, where we fit potentially ambiguous predictions to what we know happened. I'll be strict about the meaning of the prediction, as written. A prediction in a published book is a form of communication, so if Kurzweil actually meant something different to what was written, then the fault is entirely his for not spelling it out unambiguously.

One exception to that strictness: I'll be tolerant on the timeline, as I feel that a lot of the predictions were forced into a "ten years from 1999" format. So I'll estimate the prediction accurate if it happened at any point up to the end of 2011, if data is available. 

The number of predictions actually made seem to vary from source to source; I used my copy of "The Age of Spiritual Machines", which seems to be the original 1999 edition. In the chapter "2009", I counted 63 prediction paragraphs. I then chose ten numbers at random between 1 and 63, and analysed those ten predictions for correctness (those wanting to skip directly to the final score can scroll down). Seeing Kurzweil's nationality and location, I will assume all prediction refer only to technologically advanced nations, and specifically to the United States if there is any doubt. Please feel free to comment on my judgements below; we may be able to build a Less Wrong consensus verdict. It would be best if you tried to reach your own conclusions before reading my verdict or anyone else's. Hence I present the ten predictions, initially without commentary:

continue reading »