CarlShulman comments on Kurzweil's predictions: good accuracy, poor self-calibration - Less Wrong

33 Post author: Stuart_Armstrong 11 July 2012 09:55AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (39)

You are viewing a single comment's thread.

Comment author: CarlShulman 14 July 2012 10:15:47PM *  11 points [-]

You should have the predictions coded by blinded independent raters. Undergrads aren't too expensive, and you can default to Amazon's Mechanical Turk (although they may google the text) for even cheaper and faster subjects.

Also, a helpful control would be to give the independent raters predictions that resemble Kurzweil's but with the signs flipped. E.g. instead of saying "Technology X will be ubiquitous" one would say "Technology X will not be ubiquitous." When you add up the percentages who agree with each statement (with only one presented to each rater) I suspect you will get a total over 100%. Experiments with horoscopes, "psychic readers" and similar have shown like effects.

I don't think Prediction 18 is true for primary and secondary school.

The Prediction 8 "animated personality" bit is clearly wrong, and I don't think it should be cut off as 'less important' or 'trivial' if you weren't doing this elsewhere. If you're going to weight by importance then that should be done with blind rating too.

Regarding Prediction 29, scoring as "true" something developed soon after the specified date [EDIT: seems fishy to me]. When predictions are only 10 years out for something widely agreed to be inevitable, good timing is the only impressive part of a prediction.

Comment author: Stuart_Armstrong 24 July 2012 03:19:52PM 1 point [-]

That's a good idea - will do that when I get back.

Regarding Prediction 29, scoring as "true" something developed soon after the specified date contradicts your prior plan to only count successes BEFORE the predicted date. When predictions are only 10 years out for something widely agreed to be inevitable, good timing is the only impressive part of a prediction. If that is allowed to slip it undercuts the whole exercise.

I stated that I decided to be somewhat flexible on the timeline, specifically because it felt like a lot was being shoehorned into the "ten years from 1999" format.

Comment author: CarlShulman 25 July 2012 01:37:22AM 0 points [-]

Edited my comment to reflect this.