(I don't have answers to your specific questions, but here are some thoughts about the general problem.)

I agree with most of you said. I also assign significant probability mass to most parts of the argument for hope (but haven't thought about this enough to put numbers on this), though I too am not comforted on these parts because I also assign non-small chance to them going wrong. E.g., I have hope for "if AI is visible [and, I add, AI risk is understood] then authorities/elites will be taking safety measures".

That said, there are some steps in... (read more)

Which historical events are analogous to AI risk in some important ways? Possibilities include: nuclear weapons, climate change, recombinant DNA, nanotechnology, chloroflourocarbons, asteroids, cyberterrorism, Spanish flu, the 2008 financial crisis, and large wars.

Cryptography and cryptanalysis are obvious precursors of supposedly-dangerous tech within IT.

Looking at their story, we can plausibly expect governments to attempt to delay the development of "weaponizable" technology by others.

These days, cryptography facilitates international trade. It seems like a mostly-positive force overall.

One question is whether AI is like CFCs, or like CO2, or like hacking.

With CFCs, the solution was simple: ban CFCs. The cost was relatively low, and the benefit relatively high.

With CO2, the solution is equally simple: cap and trade. It's just not politically palatable, because the problem is slower-moving, and the cost would be much, much greater (perhaps great enough to really mess up the world economy). So, we're left with the second-best solution: do nothing. People will die, but the economy will keep growing, which might balance that out, because ... (read more)

Here are my reasons for pessimism:

1. There are likely to be effective methods of controlling AIs that are of subhuman or even roughly human-level intelligence which do not scale up to superhuman intelligence. These include for example reinforcement by reward/punishment, mutually beneficial trading, legal institutions. Controlling superhuman intelligence will likely require qualitatively different methods, such as having the superintelligence share our values. Unfortunately the existence of effective but unscalable methods of AI control will probably lull el

@Lukeprog, can you

(1) update us on your working answers the posed questions in brief? (2) your current confidence (and if you would like to, by proxy, MIRI's as an organisation's confidence in each of the 3:

Elites often fail to take effective action despite plenty of warning.

I think there's a >10% chance AI will not be preceded by visible signals.

I think the elites' safety measures will likely be insufficient.

Thank you for your diligence.

There's another reason for hope in this above global warming: The idea of a dangerous AI is already common in the public eye as "things we need to be careful about." A big problem the global warming movement had, and is still having, is convincing the public that it's a threat in the first place.

Who do you mean by "elites". Keep in mind that major disruptive technical progress of the type likely to precede the creation of a full AGI tends to cause the type of social change that shakes up the social hierarchy.

Combining the beginning and the end of your questions reveals an answer.

Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of [nuclear weapons, climate change, recombinant DNA, nanotechnology, chloroflourocarbons, asteroids, cyberterrorism, Spanish flu, the 2008 financial crisis, and large wars] just fine?

Answer how just fine any of these are any you have analogous answers.

You might also clarify whether you are interested in what is just fine for everyone, or just fine for the elites, or just fine for the AI in question. The answer will change accordingly.