Wisdom of the Crowd: not always so wise
I have a confession to make: I have been not "publishing" my results to an experiment because the results were uninteresting. You may recall some time ago that I made a post asking people to take a survey so that I could look at a small variation of the typical "Wisdom of the Crowds" experiment where people make estimates on a value and the average of crowd's estimates is better than that of all or almost all of the individual estimates. Since LessWrong is full of people who like to do these kinds of things (thank you!), I got 177 responses - many more than I was hoping for!
I am now coming back to this since I happened upon an older post by Eliezer saying the following
When you hear that a classroom gave an average estimate of 871 beans for a jar that contained 850 beans, and that only one individual student did better than the crowd, the astounding notion is not that the crowd can be more accurate than the individual. The astounding notion is that human beings are unbiased estimators of beans in a jar, having no significant directional error on the problem, yet with large variance. It implies that we tend to get the answer wrong but there's no systematic reason why. It requires that there be lots of errors that vary from individual to individual - and this is reliably true, enough so to keep most individuals from guessing the jar correctly. And yet there are no directional errors that everyone makes, or if there are, they cancel out very precisely in the average case, despite the large individual variations. Which is just plain odd. I find myself somewhat suspicious of the claim, and wonder whether other experiments that found less amazing accuracy were not as popularly reported.
(Emphasis added.) It turns out that I myself was sitting upon exactly such results.
The results are here. Sheet 1 shows raw data and Sheet 3 shows some values from those numbers. A few values that were clearly either jokes or mistakes (like not noticing the answer was in millions) were removed. In summary: (according to Wikipedia) 1000 million people in Africa (as of 2009) whereas the estimate from LessWrong was 781 million and the first transatlantic telephone call happened in 1926 whereas the average from the poll was 1899.
There! I've come clean!
I had deferred making this public because I thought the result that I was trying to test wasn't really being tested in this experiment, regardless of the results. The idea (see my original post linked about) was to see whether selecting between two choices would still let the crowd average out to the correct value (this two-option choice was meant to reflect the structure of some democracies). But how to interpret the results? It seemed that my selection of values is too important and that the average would change depending on what I picked even if everyone was to make an estimate, then look at the two options and choose the best one. So perhaps the only result of note here is that for the questions given, Less Wrong users were not particularly great at being a wise crowd.
Where Fermi Fails: What is hard to estimate?
I have a whimsical challenge for you: come up with problems with numerical solutions that are hard to estimate.
This, like surprisingly many things, originates from a Richard Feynman story:
One day I was feeling my oats. It was lunch time in the technical area, and I don't
know how I got the idea, but I announced, "I can work out in sixty seconds the answer to
any problem that anybody can state in ten seconds, to 10 percent!"
People started giving me problems they thought were difficult, such as integrating
a function like 1/(1 + x 4 ), which hardly changed over the range they gave me. The hardest
one somebody gave me was the binomial coefficient of x 10 in (1 + x) 20 ; I got that just in
time.
They were all giving me problems and I was feeling great, when Paul Olum
walked by in the hall. Paul had worked with me for a while at Princeton before coming
out to Los Alamos, and he was always cleverer than I was...
So Paul is walking past the lunch place and these guys are all excited. "Hey,
Paul!" they call out. "Feynman's terrific! We give him a problem that can be stated in ten
seconds, and in a minute he gets the answer to 10 percent. Why don't you give him one?"
Without hardly stopping, he says, "The tangent of 10 to the 100th."
I was sunk: you have to divide by pi to 100 decimal places! It was hopeless.(From Surely You're Joking Mr. Feynman section "Lucky Numbers")
So what would you ask Richard Feynman to solve? Think of this as the reverse of Fermi Problems.
Number theory may be a rich source of possibilities here; many functions there are wildly fluctuating, require prime factorization and depend upon the exact value of the number rather than it's order of magnitude. For example, I challenge you to compute the largest prime factor of 650238.
(My original example was: "For example, I challenge you to compute the greatest common denominator of 10643 and 15047 without a computer. This problem has the nice advantage of being trivial to make harder to compute - just throw in some extra primes." It has been pointed out that I forgot Euclid's algorithm and have managed to choose about the only number theoretic question that does have an efficient solution.)
[POLL] Wisdom of the Crowd experiment
Many of you will be familiar with the "Wisdom of the Crowd" - a phenomenon where the average result of a large poll of people's estimates tends to be very accurate, even when most people make poor estimates. I've written a short poll to test a small variant of this setup which I would like to test.
Please fill out this short poll.
Specifically, I want to see how the weighted average of the results performs when the question is posed as "is the value in question closer to A or B?" This change is inspired by your usual two-party election where people choose between two extreme values, when many voters have opinions in the middle of the two.
Thank you for helping!
I'm a little worried about anchoring in this survey. Suggestions for how to improve it would be appreciated.
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)