Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

FAWS comments on What a practical plan for Friendly AI looks like - Less Wrong

1 Post author: Mitchell_Porter 20 August 2011 09:50AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (13)

You are viewing a single comment's thread.

Comment author: FAWS 20 August 2011 12:31:41PM 10 points [-]

Simplicity has its value, but one shouldn't pursue plans any simpler than the simplest plan that might actually work. We don't know whether it's even in principle possible for us to knowingly construct an explicit goal system that matches our values and doesn't offer any room for a super-human AI to cleverly "optimize" in unexpected ways without referencing our values themselves. But I would be extremely skeptical of any attempt inspired by contract law. In your analogy this plan sounds more like "balloon that goes really high" to me. Perhaps the Singinst is pursuing Verneian cannons and no one has thought of rockets yet.