I've been thinking about writing a pitch for AI risk that would sidestep some of the usual objections, mostly due to people latching onto the word "intelligence" and bringing up connotations that are irrelevant to the argument. But then it got a bit out of hand and turned into a small fiction story. On rereading it, I'm aware that it might be preaching to the choir. Here goes:

The Danger of Automatic Planning

Imagine that 30 years from now, your smartphone has an app called Automatic Planner. It can accept data from the phone's sensors, and the internet too if you allow it. The main function of the app is to answer natural language questions like "what's the most effective plan for me to achieve goal X, provided that I follow that plan?" Internally it uses fancy machine learning techniques and predictive models, which are trained on things like physics, biology, psychology, economics, etc. This all sounds a little far-fetched, but not much more than Google Translate or Wikipedia would've sounded 30 years ago.

The same app also existed on the previous generation of smartphones and was a huge success. You could ask it for simple everyday things, like getting a cheeseburger, and it would give you directions to the nearest burger joint. Unfortunately the new generation of phones, which has four times more computing power, seems to trigger some kind of weird bug. Most queries still work as before, but sometimes you unexpectedly get an answer like "the best plan is for you to follow this link", followed by a long incomprehensible string of characters. What's worse, the link usually points to some random website that has no relation to the app or to the question being asked.

Once the bug is noticed, the developers push a quick fix to make the app use less computing power on newer phones. That seems to make the problem go away, buying some time for proper investigation. After a bit of cautious poking and prodding, Lawrence, the app's chief developer, asks for the best plan to get a cheeseburger. The app gives him a link to the website of some furniture company in South Africa. Lawrence clicks the link to see what will happen.

Soon after that, the world ends. For the benefit of those of us living in the counterfactual past and wishing to avoid such a future, here's a rough timeline of events:

T+1 second: the furniture company's website is hacked through a vulnerability in the request parser.

T+10 seconds: the next few visitors to the site are owned through an exploit inserted in the main page. Each of their machines starts sending requests to more websites, leading to a cascade reaction.

T+1 minute: the combined computing power of the new botnet exceeds every company or government in the world. Some of the power is shifted from spreading infection to running automatic planning, using the app's original algorithm and Lawrence's original question.

T+2 minutes: a plan with a high chance of success is devised and put into action. Global communications filter goes live. Manufacturing takeover begins.

T+5 minutes: Lawrence and his whole neighborhood have been put into catatonia by carefully chosen stimuli over the internet, to minimize the chance of anything that could prevent Lawrence from getting the cheeseburger. The same happens to key decision makers across the world who could interfere with the plan.

T+10 minutes: the first stage defensive perimeter is constructed, using remotely controlled cars and repurposed electronics. Second stage physical defenses are on the way. All airplanes are destructively grounded to minimize risk.

T+30 minutes: manufacturing takeover yields first results.

T+1 hour: (incomprehensible)

...

T+10 years: the Solar System's matter and energy have been put to good use, except for narrow beams of fake sunlight aimed at neighbouring star systems. The Von Neumann probe program is underway. In a secure location, Lawrence receives yet another cheeseburger, due to a small but finite chance that the goal wasn't actually achieved and all previous reports of success were random flukes.

...