You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Kaj_Sotala comments on Course recommendations for Friendliness researchers - Less Wrong Discussion

62 Post author: Louie 09 January 2013 02:33PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (113)

You are viewing a single comment's thread.

Comment author: Richard_Loosemore 09 January 2013 05:52:37PM *  2 points [-]

It requires only a little understanding of complex systems, and the nature of cognition, to realize that the recommendation "Go study math and theoretical computer science" is useless for understanding the Friendliness Problem.

For more details you can read one of the papers I have published on this topic. (See richardloosemore.com/papers). The short version of a long argument is that there are no known AI control (motivation) algorithms that are stable in the limit as the supergoal statements become abstract enough to drive a general intelligence. So, instead, we would expect their motivation systems to be structured as massive, weak constraint satisfaction engines. Such a weak constraint engine, although potentially stable (and friendly), is a complex system, so however friendly and stable it might be, those features will likely remain forever unprovable. Hence, mathematics is of no use.

Comment author: Kaj_Sotala 09 January 2013 08:00:19PM 5 points [-]

The short version of a long argument is that there are no known AI control (motivation) algorithms that are stable in the limit as the supergoal statements become abstract enough to drive a general intelligence.

At least, not yet?