In other areas we have somewhat reliable social mechanisms to validate assertions that absurd-appearing claims are in fact reasonable. So when an assertion seems absurd, and there is no social validation that it is still reasonable, we are willing to all the more conclude the claim is not reasonable. So perhaps when we consider claims about the future, we do not sufficiently correct for the fact that in this area we do not have reliable social mechanisms to validate assertions.
I think this is another key application of the way of Bayes. The usefulness of typical future predictions is hampered by the expectation of binary statements.
Most people don't make future pronouncements by making lists of 100 absurd-seeming possibilities each with a low but significant probability and say "although I would bet againt any single one of these happening by 2100, I predict that at least 5 of them will."
A classic simplified model for predicting uncertain futures is a standard tournament betting pool (like the NCAAs for instance). In any reasonably competitive 64 team field, given an even bet on the best team to be the winner, you would be against. But it is still correct to bet the best team to win in a pool (barring any information about other bets). OTOH, if you have big upset incentives, or if you know who else is betting on what, sometimes you can make profitable (+EV) bets on teams that are less likely to win than the best team, because those bets are claims of the form "I believe team X has greater than Y% probability to do Z", where Y can be arbitrarily low.
Predicting futures is similar. Presumably crazy future predictions look absurd even to field experts because they have a very low probability of occuring. It is right to bet against all of them one on one. But the number of such absurd but not impossible predictions is so large, that it is not right to bet against all of them together. As we head further into the future, the probability that some absurd thing will happen rapidly approaches 1.
The problem is figuring out which ones to bet on if you are making a typical prediction list that is phrased "In year 2100 thus and so will be the case". And the answer is that we don't have enough information to make any absurd predictions with even close to 50% confidence. If we could make a prediction of something with 50% confidence then, at least within fields possessing appropriate knowledge, it would not be considered absurd.
I'd like to see more futurists make predictions of the form I mentioned in my second paragraph, similar to Robin's approach in the list of 10 crazy things he believes.
Because if experts did that, it would get us thinking more about the 1000 or so currently foreseeable directions from which the 10-20 absurd changes of the next 100 years are most likely to come.
Eliezer:
I think that if you have a project of working through the cognitive biases for which we have evidence, considering each one separately, it is an excellent project, and likely to lead to cumulative effects on this blog, if anything is. I applaud what you are doing.
Perhaps one of the problems with predicting futures is summed up in one philosopher's version: Prediction is a tricky business, especially about the future.
Just one observation: who could have predicted in the early 1950s that the transistor would be one of the more significant culture-changing inventions? The transistor led to the transistor radio, which led to people able to listen to that radio almost anywhere (particularly at the beach, a favorite gathering spot for young people). It led to smaller and more portable electronic devices, which propelled more and more societal mobility.
The automobile is another great culture-changer, but that was perhaps more obvious to someone who really thought about it. Maybe Henry Ford was one of those.
A few people got it frightfully wrong: The IBM executive who held that the global market for big computers would be five or so. And Bill Gates, who held that 640k of RAM is plenty for anybody. Add Steve Jobs, who hed that black and white CRTs were just fine, thank you. Add the British executive, replying to one of Alexander G. Bell's telephone salesmen: "Why would we need these tele-phones? We have an ample supply of messenger boys".
There seems to be a common thread to all of those: what's working now is perfectly fine - why change? Implied is: after all, the world isn't going to change.
My wife maintains that "The Magnificent Ambersons" qualifies as a science-fiction novel, because it talks about the effects of technology (specifically, industrialization) on society (with the Ambersons a metaphor for the rest of us).
Dogbert's advice on how to (get paid to) predict the future:
Assume all positive trends will continue indefinitely. Assume all negative trends are about to end.
Another thought: People allegedly overestimate near-term changes and underestimate future changes. This could also be explained if, as suggested above, people tend to visualize a single change of the same magnitude as the largest change they remember experiencing. Over the next five years we are not likely to see the next Internet-sized change. Over the next twenty years, we are likely to see more total change than that, even if it does not stem from a single source.
On the topic of reversing changes to appreciate their absurdity: movies that were made in say the 40s or 50s, seem much more alien to me than modern movies allegedly set hundreds of years in the future, or in different universes. Watch a movie from 1950 and see how long it takes for them to show a man slapping a woman. Doesn't happen a lot in Lord of the Rings, does it?
"On the topic of reversing changes to appreciate their absurdity: movies that were made in say the 40s or 50s, seem much more alien to me than modern movies allegedly set hundreds of years in the future, or in different universes."
Most people do not know enough history (or rather, the specific parts of history) to even realize how absurd the past was. If you read a high-school level history textbook, which is the most information 99% of the public will actually remember (if that), history seems a great deal like the modern day: people had politics and governments and wars and good guys and bad guys and issues and so on, just like they do in the movies. The absurd parts get subtracted out, or added on as "irrelevant" trivia.
".. but technological change feeds on itself, and therefore has a positive second derivative."
Nitpick: If technological progress were merely quadratic in time, then too it would have a positive second derivative. Kurzweil of course claims something much stronger - that technological progress is exponential in time, which means the first derivative and all succeeding derivatives are also exponential.
There was an east-european economist (I've forgotten his name) who claimed that technological progress is exponential at some fixed rate that differs by society. His idea was that the limiting factor was the creation of prototypes. You make one prototype and learn from it, and that lets you make the next prototype, and so on.
So he said that the progress kept up at the same rate during the Depression even though the new technology didn't have much of a market. Because they could afford to keep on building prototypes all through that time. After WWII a whole lot of new technology got released to the public. And the prototype-building went on at the same old rate.
It could be true. He at least made a somewhat plausible rationale.
Part of why the future looks absurd is that people want novelty--not absolute novelty and not all the time, but a lot of smart and weird people are working on making changes, some of which will catch on. A futurist isn't going to be smart and weird enough to predict all the possible changes being offered or which ones will have a long term effect.
It's not just that technological change builds on itself, so does social change. I don't think it was completely obvious that the civil rights movement would contribute to gay marriage becoming a serious political issue.
No matter how hard you try, you are of your time. You can expand the range of your imagination, but the future outnumbers you.
I'm still working on the question of why the future isn't just unpredictable, it's absurd. Maybe there's something about human cultures which requires limiting both what people do and what people can imagine anyone doing to a small part of the range of possibilities.
Part of why the future looks absurd is that people want novelty--not absolute novelty and not all the time, but a lot of smart and weird people are working on making changes, some of which will catch on. A futurist isn't going to be smart and weird enough to predict all the possible changes being offered or which ones will have a long term effect.
It's not just that technological change builds on itself, so does social change. I don't think it was completely obvious that the civil rights movement would contribute to gay marriage becoming a serious political issue.
No matter how hard you try, you are of your time. You can expand the range of your imagination, but the future outnumbers you.
I'm still working on the question of why the future isn't just unpredictable, it's absurd. Maybe there's something about human cultures which requires limiting both what people do and what people can imagine anyone doing to a small part of the range of possibilities.
Society seems to be evolving both to AND from ridiculousness, but at the same time, some things have not changed at all. We still eat exactly the same foods as people did 2000 years ago, and we still have the exact same bodies as well as emotions as we did back then! The growth of technology has caused some societies on this earth to progress, but at the same time, the backlash against it has caused some to regress back into the ninth century. Modern society is also becoming much more perverse as a result. There's forms of sexuality out there nobody would've even conceived of before the internet came around, much less admitted to them, or practiced them in such a way that was viewable by millions of people. I think Dali invented his "cledyllism" thing as a joke, but it seems like cledyllism has taken over. We now live in a world where, as a result of sexual liberation, the women have become masculinised, and men are becoming more and more feminised, and by that I don't mean in just a touchy-feely way. I mean physically. Would they have hired a guy with shoulder-length hair, an earring in his left ear, and a t-shirt with a skull on it for any job in the 40s? Never! Or if a woman decided to shave her head and get like 50 piercings they would've sent her to the mental hospital! People are definitely becoming less conformist, and more individualist, and I think that trend will continue (transhumanism, extropianism, all that). Based on extrapolation I see the following becoming the dominant trends of the future:
Breakdown of major cities, as more people move to the country and cyber-commute from there.
Greater civility, as more people realise its not about what you know, but who you know.
Societal division being primarily on religious, as opposed to ethnic lines.
The abolition of state and local governments, the corporation as we know it, education as we know it, and the family as we know it.
If you notice, people are also getting a lot more intelligent than they were in years past, and it's not just the internet that's doing it. Just look at any movie from the 30s or 40s. There's usually just one plot line. Nowadays if you watch even a stupid mass-produced hour-long TV show, it'll have like 6 or more of them. And even little kids will understand them, and their significance. In other words, I think because we now have basically a global culture, you get a better intermixing of genes, as well as exposure to different ideas and information.
The question of IQ and its relationship to genes is not so significant as discussed in this wired article.
http://www.wired.com/wired/archive/13.05/flynn.html
That's an interesting thought -- that because the past was absurd and lead to the normal now, that people expect the future to be less absurd than today. Another example of the absurd past -- my grandmother was cured of her "evil left-handedness" habits. I predict that similar social changes -- legalization of marijuana and equal rights for gays -- are now inevitable. In the future these will be seen as normal and expected and children will be shocked to learn that it was ever otherwise, even though now many people decry these as proof of society's immorality.
However, I think that social changes are a whole different category than technological changes (at least until we decide to edit human nature).
I think describing the quoted material as "one popular entertainment was setting a cat on fire" is like saying that in our time, one popular entertainment is praying to Jesus. A religious festival is literally popular entertainment in the sense that it is done by the people, but phrasing that way implies that someone of that time would say "I think I'll burn a cat" in the same way that we'd say "I think I'll go see a movie".
Followup to: Stranger than History, Absurdity Heuristic / Absurdity Bias
Why is the future more absurd than people seem to expect? (That is: Why, historically, has the future so often turned out to be more "absurd" than people seem to have expected?)
One obvious reason is hindsight bias. Hindsight does not just cause people to severely underestimate how much they would have been surprised. Hindsight also leads people to overestimate how much attention they would have paid to the key factors, the factors that turned out to be important. As R. H. Tawney put it:
"Historians give an appearance of inevitability to an existing order by dragging into prominence the forces which have triumphed and thrusting into the background those which they have swallowed up."
When people look at historical changes and think "I could have predicted X" or "You could have predicted X if you looked at factors 1, 2, and 3"; then they forget that people did not, in fact, predict X, perhaps because they were distracted by factors 4 through 117. People read history books, see coherent narratives, and think that's how Time works. Underestimating the surprise of the present, they overestimate the predictability of the future.
I suspect that a major factor contributing to absurdity bias is that, when we look over history, we see changes away from absurd conditions such as everyone being a peasant farmer and women not having the vote, toward normal conditions like a majority middle class and equal rights. When people look at history, they see a series of normalizations. They learn the rule, "The future grows ever less absurd over time."
Perhaps one way to comprehend the bizarreness of the future would be to try and imagine historical changes occurring in reverse - how absurd would it be if all your electrical appliances suddenly disappeared, or you were transformed into a peasant farmer? Even if the future is nicer than the past, it will feel at least that absurd.
The correspondence bias of social psychology may also play a role in how we fail to learn from history - or so my own experience suggests. When we read about the strange behaviors of people in other eras, we may see them as people with a disposition to that strange behavior, rather than properly comprehending the strangeness of the times. In the 16th century, one popular entertainment was setting a cat on fire. If you think to yourself "What horrible people they must be!" then you have, to the same extent, diminished your appreciation of what horrible times they lived in.
We see at least some social and technological changes during our own lifetime. We do have some experience of genuine future shock. Why wouldn't this be enough to extrapolate forward?
According to Ray Kurzweil's thesis of accelerating change, our intuitions about the future are linear - we expect around as much change as occurred in the past - but technological change feeds on itself, and therefore has a positive second derivative. We should expect more technological change in the future than we have seen in the past, and insofar as technology drives cultural change, we should expect more cultural change too.
Or that, in my opinion, is the strongest version of Kurzweil's theory that can be put forward. Kurzweil dwells on Moore's Law and smoothly predictable exponential curves, but this seems to me both iffy and unnecessary. A curve does not need to be smooth or exponential to have a positive second derivative. And our cultural sensitivity to, say, computing power, is probably logarithmic anyway, obeying Weber's Law - a 20% increase in computing power probably feels the same whether it's from 1MHz to 1.2MHz, or 2GHz to 2.4GHz. In which case, people extrapolating the future "linearly" should get it pretty much correct.
But if you pull back and view the last few millennia, not just the last few decades, the strength of the core idea becomes obvious - technology change does feed on itself and therefore does speed up.
I would actually question Kurzweil's assertion that people extrapolate the past linearly into the future. Kurzweil may be too optimistic here. As discussed earlier, dwellers on flood plains do not extrapolate from small floods to large floods; instead, small floods set a perceived upper bound on risk. I suspect that when people try to visualize the strangeness of the future, they focus on a single possible change, of no greater magnitude than the largest single change they remember in their own lifetime.
The real future is not composed of single developments, but many developments together. Even if one change can pass the futurism filter, to suppose three absurdities simultaneously - never mind twenty - would entirely overload the absurdity meter. This may also explain why future projections get wronger and wronger as they go further out. People seem to imagine futures that are minimally counterintuitive, with one or two interesting changes to make a good story, rather than a realistic number of changes that would overload their extrapolation abilities.
What other biases could lead us to underestimate the absurdity of the future?