Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Danielle_Fong comments on Growing Up is Hard - Less Wrong

28 Post author: Eliezer_Yudkowsky 04 January 2009 03:55AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (41)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Danielle_Fong 04 January 2009 05:47:37PM 0 points [-]

"You could build a modular, cleanly designed AI that could make a billion sequential upgrades to itself using deterministic guarantees of correctness."

Really? Explain how? It seems like a general property of an intelligent system that it can't know everything about how with would react to everything. That falls out of the halting theorem (and for that matter Godel's first incompleteness theorem) fairly directly. It might be possible to make a billion sequential upgrades with probabilistic guarantees of correctness, but only in a low entropy environment, and even then it's dicey, and I have no idea how you'd prove it.