You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

jacob_cannell comments on Could you tell me what's wrong with this? - Less Wrong Discussion

1 Post author: Algon 14 April 2015 10:43AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (64)

You are viewing a single comment's thread. Show more comments above.

Comment author: jacob_cannell 14 April 2015 05:24:58PM *  0 points [-]

An AGI also has the advantage of being able to duplicate instances. If you would create 1000000 copies of the 10 best computer security researchers, that army could likely hack itself into every computer system available.

Just a little nitpick here, but your last statement is unlikely. There are actual computer systems that are completely unhackable even by any team of skilled security researchers (non-networked devices, etc).

Of course, the generalization of your point is much more likely, but the likely forms of AI takeover probably involve a diverse set of skills, including subtle and complex strategic social/cultural manipulations. Human brains are more hackable than many computer systems in this regard.

Comment author: ChristianKl 14 April 2015 09:55:42PM 1 point [-]

Just a little nitpick here, but your last statement is unlikely.

I did use the word "available" as a qualifier. Some airgapped computers don't qualify as available. But 100% isn't required.

I consider it politically impossible to shut down all nonairgapped computers. Taking away computers creates civil war and in civil war computers are useful to win conflicts. The AGI just has to stay alive and self improve.

For an AGI hacking a great number of computers allows the AGI to use those computers to

Comment author: MrMind 15 April 2015 07:35:32AM 0 points [-]

Just a little nitpick here, but your last statement is unlikely. There are actual computer systems that are completely unhackable even by any team of skilled security researchers (non-networked devices, etc).

Only if you limit 'hacking' to networking and don't include things like social engineering, blackmailing and physical attack/intrusion.