For the arrival of AGI the answer to your last questions is YES, YES, YES.
You should be very skeptical - as I was when I first encountered the rationalists.
Yet the evidence that AGI will likely happen in our lifetime is overwhelming.
Suspiciously so.
Sometimes extraordinary things do happen. Sometimes dozens bits of evidence is right in front of your lieing eyes. We are already living in an extraordinary time. You only have to see what's right in front of you.
I suspect anthropics but chalk it up to whatever god you pray to.
Actually, lesswrong AGI warnings don't sound like they could be the plot of a successful movie. In a movie, John Connor organizes humanity to fight against skynet. That does not seem plausible with LW-typical nanobot scenarios.
To me, nanobots don't them like the are central to LW stories about AI risk.
If you would ask people on an LW census "If AI causes the extinction of humans, how likely do you think that nanobots play a huge part in that", I would expect the median percentage to be single digits or lower.
I agree that Nanobots are not a necessary part of AI takeover scenarios. However, I perceive them as a very illustrative kind of "the AI is smart enough for plans that make resistance futile and make AI takeover fast" scenarios.
The word "typical" is probably misleading, sorry; most scenarios on LW do not include Nanobots. OTOH, LW is a place where such scenarios are at least taken seriously.
So p(scenario contains Nanobots|LW or rationality community is the place of discussion of the scenario) is probably not very high, but p(LW or rationality community is the place of discussion of the scenario|scenario contains Nanobots) probably is...?
This seems pretty false. There is at least one pretty successful fiction book written about the intelligence explosion (which, imo, would have been better if in subsequent books gur uhznaf qvqa'g fheivir).
Gnargh. Of course someone has a counterexample. But I don't think that is the typical lw AGI warning scenario. However, this could become a "no true Scotsman" discussion...
You mean like Gwern's It Looks Like You’re Trying To Take Over The World? I think that made a good short story. Though I don't think it would make a good movie, since there's little in the way of cool visuals.
Greg Egan's Crystal Nights is also more similar to the usual way things are imagined, though uhznavgl vf fnirq ol gur hayvxryl qrhf rk znpuvan bs vg orvat rnfvre sbe gur fvzhyngrq pvivyvmngvba gb znxr n cbpxrg qvzrafvba guna gnxr bire gur jbeyq.
Crystal Nights is also very similar to Eliezer's That Alien Message / Alicorn's Starwink.
Edit: There are also likely tons more such books written by Ted Chiang, Vernor Vinge, Greg Egan, and others, which I haven't read yet so can't list with confidence and without spoilers to myself.
Thanks for the list! Yes, it is possible to imagine stories that involve a superintelligence.
I could not imagine a movie/successful story where everybody is killed by an AGI within seconds because it has prepared that in secrecy and nobody realized it, and nobody could do anything about it. Seems like lacking a happy end and even a story.
However, I am glad to be corrected, and will check the links, the stories will surely be interesting!
Ted Chiang's Chrystal Nights
Minor corrections: “Crystal Nights” does not have an H in the first word and is by Greg Egan. (The linked copy is on his own website, in fact, which also includes a number of his other works.)
Thanks! I remember consciously thinking both those things, but somehow did the opposite of that.
Nanobots destroying all humans at once are indeed poor sci-fi. But how much of this story's popularity hinges on it happening within our lifetimes?
I don't understand this question. Why would the answer to that question matter? (In your post, you write "If the answer is yes to all of the above, I’d be a little more skeptical.") Also, the "story" is not really popular. Outside of LessWrong discussions and few other places, people seem to think that every expectation about the future that involves a superintelligent agentic AGI sounds like science fiction and therefore does not have to be taken seriously.
Sorry for not being clear. My question was whether LW really likes the nanobot story because we think it might happen within our own lifetimes. If we knew for a fact that human-destroying-nanobots would take another 100 years to develop, would discussing them still be just as interesting?
Side note: I don't think the "sci-fi bias" concept is super coherent in my head, I wrote this post as best as I can, but I fully acknowledge that its not fully fleshed out.
Yes, people care about things that are expected to happen today rather than in 1,000 years or later. That is a problem that people fighting against climate change have been pointing out for a long time. At the same time, with respect to AI, my impression is that many people do not react to developments that will quickly have strong implications, while some others write a lot about caring about humanity's long-term future.
Thank you! Here's the transcript: https://www.lesswrong.com/posts/4kphivjxngJmEdWsN/transcript-tyler-cowen-on-stories
If the answer is yes to all of the above
Point 2 needs rephrasing.
"Does it sound exciting or boring?" "Yes"
This link is also relevant, and BTW, this goes for why I'm a priori skeptical of misaligned AI, even without evidence against it, because it fits way too well with story logic, and in particular one thing that misaligned AIs provide in terms of story value is an immediate conflict, whereas safe and/or aligned AIs provide less conflict and story opportunities.
https://www.understandingai.org/p/predictions-of-ai-doom-are-too-much
There are other ways to be skeptical about fictional paradigms regarding AI. For example, a common paradigm is that AIs escape human control, and then there is a long struggle. An alternative paradigm is that once conflict emerges, the AIs win quickly and humans are permanently marginalized thereafter.
And even when the AGI does do work (The Doctor), it’s been given human-like emotions. People don’t want to read a story where the machines do all the work and the humans are just lounging around.
I am taking the opportunity to recommend the culture by Ian M. Banks here is a good entry point to the series, the books can be read in almost any order. It's not like they find no space for human-like actors, but I still think these books show -by being reasonably popular- that there is an audience for stories about civilizations where AGI does all the work.
Of course, your original point still stands if you say "most people" instead.
We want to know the future, so we like reading predictions. These predictions can significantly vary in how long their time horizon is:
There’s three elements in how predictions are evaluated by society:
We all like a good story
Imagine the year is 2011 and you’re writing a movie about a future global pandemic. You hire a scientist known for helping eradicate smallpox (Larry Brilliant) as your consultant and overall try to make your story somewhat plausible. You title the movie ‘Contagion’ and release it with the following plot elements:
Nine years later a global pandemic happens in real life but (fortunately) the plotline is much more mundane:
‘Contagion’ made $136.5m in the box office back in the day. How much would it have made if was released with the same plotline as our 2020 pandemic? Probably a lot less. A movie where Paltrow sits at home coughing for a week or two and then goes back to work just doesn’t sound as exciting. Some other examples of this include:
We want to live in modestly interesting times
Terry Pratchet’s had a funny quote about a phrase (incorrectly) attributed to ancient Chinese writings:
Does this mean we want to live in “boring” times? Imagine if an omniscient seer told you at the age of 18 that absolutely nothing “interesting” would happen in the next 100 years. Even if you were born in a highly prosperous city in Switzerland, would such knowledge make you happier? I would argue that most of us are hoping to live in reasonably interesting times, which biases us towards predictions that promise change rather than stagnation.
We want interesting things to happen during our lifetimes
Quite a few of people are interested in working on AI safety. But lets once again imagine we have access to an all-powerful seer who tells us that AGI will not be developed before the year 2150. How many people would still be interested in working on AI safety then? Probably a lot less. The same deal applies to every area of human innovation: Elon Musk wants to personally witness humans landing on Mars, quantum computing researches hope to see a quantum computer built, fusion companies want to start generating power now rather than in 100 years.
And this isn’t restricted to scientists, entrepreneurs and engineers. Ukrainians want their territory liberated now rather than in 50 years. Americans worry about their personal Social Security checks much more than they worry about future fiscal troubles. Climate change advocates sound a lot more convincing if they promise something dramatic to happen now rather than sometime in the next century.
To summarize, I’d like to propose a three-prong test for judging predictions for how much science fiction bias they could possibly have:
If the answer is yes to all of the above, I’d be a little more skeptical. And conversely when writing predictions, I’d try to judge whether I’m trying to write a good story or trying to actually predict the future. Alas, reality is often disappointing.