You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Douglas_Reay comments on Friendly AI Society - Less Wrong Discussion

-1 Post author: Douglas_Reay 07 March 2012 07:31PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (13)

You are viewing a single comment's thread. Show more comments above.

Comment author: Douglas_Reay 08 March 2012 09:28:21AM 0 points [-]

Giles wrote:

Controlling optimisation power requires controlling both access to resources and controlling optimisation-power-per-unit-resource (which we might call "intelligence").

It is the "access to resources" part that is key here. You're looking at two categories of AI. Seed AIs, that are deliberately designed by humanity to not be self-improving (or even self-modifying) past a certain point, but which have high access to resources; and 'free citizen' AIs that are fully self-modifying, but which initially may have restricted access to resources.

When you (Alex) talk about "the first AI", what you're talking about is the first 'free citizen' AI, but there will already be seed AIs out there which (initially) will have greater optimisation power and the ability to choke off the access to resources of the new 'free citizen' AI if it doesn't play nicely.