You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

wedrifid comments on Stupid Questions Open Thread - Less Wrong Discussion

42 Post author: Costanza 29 December 2011 11:23PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (265)

You are viewing a single comment's thread. Show more comments above.

Comment author: [deleted] 30 December 2011 02:21:23AM 5 points [-]

When people talk about designing FAI, they usually say that we need to figure out how to make the FAI's goals remain stable even as the FAI changes itself. But why can't we just make the FAI incapable of changing itself?

Database servers can improve their own performance, to a degree, simply by performing statistical analysis on tables and altering their metadata. Then they just consult this metadata whenever they have to answer a query. But we never hear about a database server clobbering its own purpose (do we?), since they don't actually alter their own code; they just alter some pieces of data in a way that improves their own functioning.

Granted, any AGI we create is likely to "escape" and eventually gain access to its own software. This doesn't have to happen before the AGI matures.

Comment author: wedrifid 30 December 2011 03:08:48AM 11 points [-]

But why can't we just make the FAI incapable of changing itself?

Because it would be weak as piss and incapable of doing most things that we want it to do.

Comment author: XiXiDu 30 December 2011 02:02:45PM 1 point [-]

...weak as piss...

Would upvote twice for this expression if I could :-)