18 people initially volunteered to do varying amounts of assessment of Kurzweil's predictions; 9 ultimately did so.
Not uncommon.
My initial impression was that the volunteer completion rate would be higher among a group like LW members. But now I realize that was a naive assumption to make.
Is 50% lower than usual? My intuition says the norm is between 15% and 40%, with ~60% confidence.
Givewell's volunteer failure rate was apparently ~80%: http://blog.givewell.org/2011/07/13/a-good-volunteer-is-hard-to-find/ (LW discussion).
Email is a horrible method of communication, and one of the things it's horrible at is getting people to do things. This is the price to pay for how convenient it is to send an email. No doubt requiring that volunteers make a phone call to offer to volunteer instead would decrease the attrition rate but it's unclear if you'd actually get more useful work that way.
A possible comparison group to act as a control for Kurzweil's predictions is Joseph Mcmoneagle remote viewing work. His book, "The Ultimate Time Machine: A Remote Viewer's Perception of Time, and Predictions for the New Millennium" offers multiple, precise predictions that are precise enough to use your 5 point scale.
For example, on page 247:
1) By 2010, a single light fiber, half the diameter of a human hair, will be capable of carrying a million gigabits per second 2) Hard disk computer storage systems will be replaced in 2008 with electromagnetic/chemical storage systems
page 248:
3) The standard RAM on the average machine in 2008 will be 128 MB, with 256 MB the industrial standard on business machines
page 249:
4) A new home sound system will be unveiled by one of the leaders in the sound industry between 2002 and 2004. It will produce sound that is controlled by computer to simulate 360 degree surround sound, or 3-D sound simulation. The computer will use sensors to detect how many people are in the room and where they are located in comparison to the shape of the room and the furniture in it. It will then alter the sound being emitted from ten or more speakers... (continues) 5) Within ten years (1998 to 2008) there will be a silver bullet cure for most cancers, as well as the development of a vaccine for AIDS
There are many other categories of prediction in the book besides technology, including politics and government, the environment, economics, social (anthropology, archeology, arts, education, etc) and economics.
As a bonus, the predictions include the year 3000. This is nice because kurzweil loves to compare the intuitive linear view to the his law of accelerating returns. If parapsychology is like the control group for science, why not put this declaration to the test? There are many predictions in the book, probably well over 100 that can be tested already.
Granted, this book is in a popular format, but so is Kurzweil's books, so I believe it is an apt comparison. In other words, this book does not represent the absolute highest quality possible for remote viewing work. For that, you might try the Farsight Institute. Here is an example of what I would consider high quality remote viewing work:
I did not get a chance to download the data that was encrypted in order to prevent cheating. What I would like to know is if anyone on the forum knows anyone who did get a chance to snag the data prior to the remote viewing session. Furthermore, anyone else know of a way that they may be cheating or fooling themselves?
A major advantage of using this book is that it was published prior to Kurzweil's book, in 1998. But one may have to check kurzweilai.net (and Kurzweil's previous work) to see if Mcmoneagle copied (intentional or not) Kurzweil's views.
This post goes along with this one, which was merely summarising the results of the volunteer assessment. Here we present the further details of the methodology and results.
Kurzweil's predictions were decomposed into 172 separate statements, taken from the book "The Age of Spiritual Machines" (published in 1999). Volunteers were requested on Less Wrong and on reddit.com/r/futurology. 18 people initially volunteered to do varying amounts of assessment of Kurzweil's predictions; 9 ultimately did so.
Each volunteer was given a separate randomised list of the numbers 1 to 172, with instructions to go through the statements in the order given by the list and give their assessment of the correctness of the prediction (the exact instructions are at the end of this post). They were to assess the predictions on the following five point scale:
They assessed a varying amount of predictions, giving 531 assessments in total, for an average of 59 assessments per volunteer (the maximum attempted was all 172 predictions, the minimum was 10). They generally followed the randomised order correctly - there were three out of order assessments (assessing prediction 36 instead of 38, 162 instead of a 172, and missing out 75). Since the number of errors was very low, and seemed accidental, I decided that this would not affect the randomisation and kept those answers in.
The assessments (anonymised) can be found here.
In parallel, volunteers on Youtopia were also given the task of assessing the predictions. They were given the same instructions (minus the 5th and 7th clause), except that they were free to work on whichever predictions they wanted to, with the proviso that they didn't overwrite someone else's assessments. Instead, they could post a second opinion (not necessarily different from the first) in a separate column.
For some reason, prediction number 20 ("LUIs are frequently combined with animated personalities") was left out of the Youtopia assessment. In total, 204 assessments were made (171 primary assessments, 33 second opinions).
Instructions
The instructions given to the assessors were as follows:
1) The timeline of Kurzweil's prediction is up to 2011, and the location (unless specified otherwise) is the United States.
I've given Kurzweil a two-year grace period (he said they would all be true by 2009). This is because I think he forced a lot of predictions into the "true by ten years from now (1999)" format. Also, this makes it a bit easier for you, as you don't need to go as far back into history.
2) A prediction is something that allows you to make a profit.
This is the true test of a prediction: if you're in 1999, and you believe one of Kurzweil's predictions, could you make your life better than someone who didn't believe the prediction? If Kurzweil made a brilliant, correct prediction but nobody at the time would have realised what it meant, then it doesn't count as a correct prediction. A prediction needs to make sense ahead of time, in a way you can take advantage of.
3) Resolving unclear terms is maybe the most important part of your job.
Some of Kurzweil's predictions are ambiguous, including terms like "many", "most", "routinely". Figure out what these terms mean for you. Predictions are acts of communication; they are only valid if the reader understands them correctly. The truth of "people will routinely use mobile phone" depends entirely on what meaning you give to "routinely". In 1999, how you would have imagined a future in which people "routinely" use mobile phones?
4) No gain from ambiguity, no "benefit of the doubt".
Do not interpret an ambiguous statement as true, simply to give the predictor the benefit of the doubt. With hindsight, some things will seem a lot more inevitable than they actually were, and some predictions will seem as if they "must refer to X". For instance, "Two mighty towers will fall" seems like it refers to the Sep 11 terrorist attacks - but there are many ways of interpreting that figuratively or literally (two institutions will be undermined, Tolkein's "the two towers" will be made into a film, etc...). Again the question is whether people in 1999 could have foreseen something like that outcome, based on that prediction.
5) Answer the prediction you're working on, not the nearby ones.
The predictions just before and after are useful to give some context to the prediction you're currently working on, to explain some terms and clarify what Kurzweil is talking about. But you should only answer the exact prediction you're working on (don't worry, those other prediction will have someone else working on them!). Thus the second prediction in "There will be a terrorist attack on the 9th of September, 2011. 4006 people will be killed in it" is false.
6) No penalty for triviality.
In hindsight, many prediction may seem trivial or obvious. This doesn't mean they were trivial at the time. But in any case, your job is not to estimate how useful or hard the predictions were, but how accurate. "Computers will get faster" is a true prediction.
7) Follow the order in the text file included.
You are welcome and encouraged to answer more predictions than you promised to - but stick to the predictions given in the randomized text file! This will make the experiment statistically significant... Unless of course you intend to answer all the predictions!