utilitymonster comments on What is the best compact formalization of the argument for AI risk from fast takeoff? - Less Wrong

11 Post author: utilitymonster 13 March 2012 01:44AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Sort By: Best

You are viewing a single comment's thread.

Comment author: utilitymonster 14 March 2012 08:20:36PM 1 point [-]

I prefer this briefer formalization, since it avoids some of the vagueness of "adequate preparations" and makes premise (6) clearer.

At some point in the development of AI, there will be a very swift increase in the optimization power of the most powerful AI, moving from a non-dangerous level to a level of superintelligence. (Fast take-off)
This AI will maximize a goal function.
Given fast-take off and maximizing a goal function, the superintelligent AI will have a decisive advantage unless adequate controls are used.
Adequate controls will not be used. (E.g. Won’t box/boxing won’t work)
Therefore, the superintelligent AI will have a decisive advantage
Unless that AI is designed with goals that stably and extremely closely align with ours, if the superintelligent AI has a decisive advantage, civilization will be ruined. (Friendliness is necessary)
The AI will not be designed with goals that stably and extremely closely align with ours.
Therefore, civilization will be ruined shortly after fast take-off.

Comments (20)