Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Checking Kurzweil's track record

12 Post author: Stuart_Armstrong 30 October 2012 11:07AM

Predictions are cheap and easy; verification is hard, essential, and rare. For things like AI, we seem to be restricted to nothing but expert predictions - but expert predictions on AI are not very good, either in theory or in practice. If we are some experts who stand out, we would really want to identify them - and there is nothing better than a track record for identifying true experts.

So we're asking for help to verify the predictions of one of the most prominent futurists of this century: Ray Kurzweil, from his book "The Age of Spiritual Machines". By examining his predictions for times that have already come and gone, we'll be able to more appropriately weight his predictions for times still to come. By taking part, by lending your time to this, you will be directly helping us understand and predict the future, and will get showered in gratitude and kudos and maybe even karma.

I've already made an attempt at this (if you are interested in taking part in this project, avoid clicking on that link for now!). But you cannot trust a single person's opinions, and that was from a small (albeit random) sample of the predictions. For this project, I've transcribed his predictions into 172 separate (short) statements, and any volunteers would be presented with a random selection among these. The volunteers would then do some Google research (or other) to establish whether the prediction had come to pass, and then indicate their verdict. More details on what exactly will be measured, and how to interpret ambiguous statements, will be given to the volunteers once the project starts.

If you are interested, please let me know at stuart.armstrong@philosophy.ox.ac.uk (or in the comment thread here), indicating how many of the 172 questions you would like to attempt. The exercise will probably happen in late November or early December.

This will be done unblinded, because Kurzweil's predictions are so well known that it would be infeasible to find large numbers of people who are technologically aware but ignorant of them. Please avoid sharing your verdicts with others; it is entirely your own individual assessment that we are interested in having.

Comments (53)

Comment author: selylindi 30 October 2012 01:12:39PM 8 points [-]

This will be done unblinded, because Kurzweil's predictions are so well known that it would be infeasible to find large numbers of people who are technologically aware but ignorant of them.

Is this true? It could be, or alternatively it could simply appear true from your perspective of familiarity. I'm only vaguely aware of Kurzweil and have never heard any mention of him among my group of largely grad student / geek friends.

Comment author: gwern 30 October 2012 02:48:35PM 6 points [-]

I don't think it's true. I think it would be pretty easy to survey people and include some questions checking for technological awareness and Kurzweil awareness, and quietly discarding any results from people low on the former or high on the latter.

I mean, you could do it on Amazon Mechanical Turk! Such people are pretty technically sophisticated to be on Mechanical Turk in the first place, psychologists use it for surveys all the time, and a dismayingly large fraction of Turkers have college educations. It'd work fine.

Comment author: Stuart_Armstrong 31 October 2012 09:21:34AM 1 point [-]

We'll look into the Mechanical Turk angle...

Comment author: sheeplearepeople 28 November 2012 06:25:12PM 0 points [-]

http://www.kurzweilai.net/predictions/download.php

This contains all his predictions, it shouldn't be hard to verify them.

Comment author: Stuart_Armstrong 29 November 2012 11:35:10AM 0 points [-]

Self-assessment isn't a good idea. And I'm really not impressed by that one (though I shouldn't comment directly at this stage).

But we will see. I'll wait till I have all the data in.

Comment author: DaFranker 30 October 2012 02:27:11PM *  3 points [-]

I'm likewise vaguely aware of Kurzweil's reputation and accomplishments, and vaguely recall mention of a big prediction in the field of AI, a very optimistic one according to critics IIRC.

That's about the extent of it. I wasn't aware he had publicly made other predictions.

Problem is, how would one go about verifying this? Not to mention that if I wasn't already primed to not click the link, I would probably have immediately searched for the predictions in question just to know the subject under discussion.

On that note, thanks and good move Stuart, warning us beforehand about the spoiler/unblinding link.

Comment author: Nornagest 31 October 2012 12:24:33AM *  1 point [-]

I doubt it's true. I think it would be relatively easy to find technically sophisticated people who're unaware of Kurzweil's specific predictions; it'd be harder to find technically sophisticated people who're consistently unaware of his general thesis, but I'll bet you could still do it. You'd just need to look outside the transhumanist/singularitarian/AI enthusiast cluster.

Since those clusters are pretty tightly grouped in terms of conceptual underpinnings, it should be easy to filter them from a sample. Getting a good sample would be harder -- LW wouldn't do it, and personal blogs wouldn't either. Gwern's idea below looks promising but I have no idea how you'd go about it.

Comment author: Stuart_Armstrong 30 October 2012 03:17:16PM 1 point [-]

Interesting. My feeling was that it was hard to conceal that it was Kurzweil, and that people would certainly see that it was Kurzweil while googling their answers (also, we would get more interest with a semi-famous name).

Comment author: blashimov 30 October 2012 03:57:00PM 4 points [-]

I will do 20.

Comment author: Stuart_Armstrong 30 October 2012 04:04:21PM 0 points [-]

Thanks!

Comment author: orangecat 04 November 2012 02:23:07AM 3 points [-]

I'll do 10. Agreed with satt that having multiple raters for each prediction would be helpful. I previously read your previous post with the randomly selected predictions, which hopefully isn't disqualifying.

Comment author: Stuart_Armstrong 09 November 2012 10:09:37AM 1 point [-]

Cheers!

previously read your previous post with the randomly selected predictions, which hopefully isn't disqualifying.

Fine as long as you try and ignore them :-)

Comment author: MTGandP 02 November 2012 05:24:54AM 3 points [-]

I will do 10 predictions. Email mtgandp gmail.

Interestingly, Stuart seems to have adopted a similar approach to the synagogue described in Why Our Kind Can't Cooperate, with people publicly declaring their commitments.

Comment author: Stuart_Armstrong 02 November 2012 10:09:46AM 1 point [-]

Cheers!

Interestingly, Stuart seems to have adopted a similar approach to the synagogue described in Why Our Kind Can't Cooperate, with people publicly declaring their commitments.

I wish I could take credit for that level of machiavellianism...

Comment author: EricHerboso 31 October 2012 01:08:01AM 3 points [-]

I'll commit to doing 20 questions.

Comment author: Stuart_Armstrong 31 October 2012 09:20:08AM 0 points [-]

Thanks!

Comment author: SimonF 05 November 2012 02:10:46PM 2 points [-]

I will do 20, too!

Comment author: Stuart_Armstrong 09 November 2012 10:10:53AM 0 points [-]

Cheers!

Comment author: MaoShan 03 November 2012 03:26:48AM 2 points [-]

I will do ten questions. I have no bias for or against Kurzweil, and I will try my best to find out whether the prediction was accurate, regardless of who predicted it. Maybe including a few placebo questions would be a good idea. kemirunda@hotmail

Comment author: Stuart_Armstrong 03 November 2012 10:18:11AM 0 points [-]

Thanks!

Comment author: MaoShan 28 November 2012 03:08:07AM 2 points [-]

I have my ten done, but I'd like to continue down the randomized list. Is there a deadline that you would like these returned by?

Comment author: Stuart_Armstrong 28 November 2012 05:08:52PM 1 point [-]

Great! :-) Can you get them done by the 8th of December, so I can present the results at the AGI-impacts conference if need be?

Comment author: MaoShan 29 November 2012 04:20:24AM 2 points [-]

Sure. I'll send whatever I've completed by Sunday night, so you have a chance to analyze your data in time for the conference. I expect to have about fifty done by then.

Comment author: Stuart_Armstrong 29 November 2012 11:32:02AM 1 point [-]

You're being a star!

Comment author: DaFranker 02 November 2012 01:46:47PM 2 points [-]

I'd be willing to do around 10-15, whichever works to fit the number of questions to the number of people.

Comment author: Stuart_Armstrong 02 November 2012 03:44:57PM 0 points [-]

Cool, thanks!

Comment author: MrMind 31 October 2012 06:08:32PM *  2 points [-]

Sounds like fun! I'll commit to 10 predictions. Are those in a form which can be clearly examined as "correct/incorrect"?
Email at ikstef at gmail dot com

Comment author: Stuart_Armstrong 01 November 2012 10:24:40AM 1 point [-]

Cheers! There will be partial correct/incorrect scores available as well.

Comment author: bsterrett 31 October 2012 05:45:54PM 2 points [-]

I'll do 10.

What is the error-checking process? Will we fix any mistakes in our verdicts via an LW discussion after they have been gathered?

Comment author: Stuart_Armstrong 31 October 2012 05:57:02PM 1 point [-]

Thanks! I'll think about the error checking process; certainly there is some possibility for factual errors, but most of the work is in interpreting qualifiers like "ubiquitous" and "most" and mapping that to what happened in the world.

Comment author: satt 03 November 2012 05:32:39PM 2 points [-]

One way to gauge how reliable people's judgements are: have multiple people rate each Kurzweil prediction and see how well their ratings agree. So far LWers have committed to checking at least 200 predictions, so if everyone pulls through you'll be able to get multiple ratings of at least 28 questions. Those multiple ratings could then be cross-checked for each question.

(I won't volunteer to rate any statements myself because (1) I'm lazy; (2) I already have a mildly negative view of Kurzweil's predictive ability, which might make me biased; and (3) I read your earlier post and re-rated the 10 Age of Spiritual Machines predictions in that post myself, so I've already been primed in that respect.)

Comment author: Unnamed 04 November 2012 02:59:33AM 3 points [-]

One way to gauge how reliable people's judgements are: have multiple people rate each Kurzweil prediction and see how well their ratings agree.

This is a good idea. It's standard operating procedure (for measures which require a rater's judgment) to have 2 raters for at least some of the items, and to report the agreement rate on those items ("inter-rater reliability"). Be sure to vary which raters are overlapping; for example, don't give gwern and bsterrett the same 10 predictions (instead have maybe one prediction that they both rate, and one where bsterrett & Tenoke overlap, etc.) - that way the agreement rate tells you something about how much agreement there is between all of the raters (and not just between particular pairs of raters).

In cases where the 2 raters disagree, you could just have a 3rd rater rate it and then go with their rating, or you could do something more complicated (like having the two raters discuss it and try to reach a consensus).

Comment author: Tenoke 31 October 2012 03:22:44PM 2 points [-]

I can commit to 20 as well. For the record, I reckon around 3/4 of his predictions will pass.

Comment author: Stuart_Armstrong 31 October 2012 05:16:06PM 2 points [-]

Thanks!

Comment author: gwern 31 October 2012 01:19:28AM 2 points [-]

I'd be fine with 10, at least initially; email is gwern0@gmail.com if that's how you're doing it.

Comment author: gwern 22 November 2012 06:42:07PM 4 points [-]

Update: I've done 20 predictions. Not that tedious, kinda interesting actually.

Comment author: Stuart_Armstrong 31 October 2012 09:20:16AM 0 points [-]

Thanks!

Comment author: quinox 30 October 2012 11:12:24PM 2 points [-]

Sign me up for ~2 hours of questions. BTW, I'm not familiar with Kurzweil's predictions yet (I'll wait with that until after I've done your questions)

Comment author: Stuart_Armstrong 31 October 2012 09:21:07AM 0 points [-]

Thanks!

Comment author: MileyCyrus 30 October 2012 12:06:42PM 3 points [-]

I'll do 20.

Comment author: Stuart_Armstrong 30 October 2012 03:12:44PM 1 point [-]

Thanks!

Comment author: Eneasz 30 October 2012 07:30:09PM 1 point [-]

Before committing to a number, how long do you think it would take to research a given question? I can't do very many if they're likely to take an hour each.

Comment author: Stuart_Armstrong 31 October 2012 05:20:00PM 1 point [-]

The questions will be grouped together (they are constructed grammatically so that they follow on from each other), so you'll generally treating 4-5 questions as a single block. It took me a a few hours to do my research; I did ten blocks, which I think were about 20-30 predictions.

Comment author: Eneasz 31 October 2012 07:52:02PM *  2 points [-]

Alrighty, put me down for 10.

ETA: Forgot email. :) embrodski at gmail

Comment author: Stuart_Armstrong 01 November 2012 10:23:56AM 1 point [-]

Cheers!

Comment author: [deleted] 02 November 2012 01:36:15PM 0 points [-]

I'm up for that -- andrew @ thenationalpep.co.uk . I can probably do ten. Disclosure of biases upfront -- I'm not familiar with that specific book of Kurzweil, or with much of Kurzweil's work in general, so I don't know what his predictions are. But I am familiar with his book The Singularity Is Near, which I thought utterly, comprehensively wrongheaded to a point where most of what I could say about it would seem like personal abuse, so I suspect that a relatively low proportion of his predictions will come true.

Comment author: Stuart_Armstrong 02 November 2012 03:45:26PM 1 point [-]

Cool, thanks!

Comment author: Dallas 31 October 2012 02:27:31AM 0 points [-]

I will examine 30 questions. dallasjhaugh at gmail dot com

Comment author: Stuart_Armstrong 31 October 2012 09:20:51AM 0 points [-]

Thanks!

Comment author: baiter 01 November 2012 12:59:45PM 1 point [-]

I would do 10.

Comment author: Stuart_Armstrong 01 November 2012 01:14:40PM 0 points [-]

Cheers!

Comment author: lavalamp 30 October 2012 11:35:24PM 1 point [-]

I'm aware of Kurzweil but I don't think I've read anything of his except for your earlier fact checking article (and that was long enough ago that I don't remember a thing about it).

It wouldn't be so hard to add a "do you think this prediction was made by Kurzweil or someone else? (don't google, but if you accidentally found the answer while researching, don't pretend not to know, either)" checkbox.

Comment author: Stuart_Armstrong 31 October 2012 09:25:34AM 2 points [-]

We'll ask about their familiarity with Kurzweil in general - but remember that the volunteers are going to do research on the questions, so it's very likely they'll realise it's Kurzweil at that point.