John_Maxwell_IV comments on Q&A with new Executive Director of Singularity Institute - Less Wrong

26 Post author: lukeprog 07 November 2011 04:58AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (177)
Tags: community

Comments (177)

You are viewing a single comment's thread. Show more comments above.

Comment author: John_Maxwell_IV 23 March 2012 06:51:19AM 3 points [-]

From where I'm sitting, proving an AGI design Friendly seems even more difficult and error-prone than proving a crypto scheme secure, probably by a large margin, and there is no decades of time to refine the proof techniques and formalizations.

Correct me if I'm wrong, but it doesn't seem as though "proofs" of algorithm correctness fail as frequently as "proofs" of cryptosystem unbreakableness.

Where does your intuition that friendliness proofs are on the order of reliability of cryptosystem proofs come from?

Comment author: Wei_Dai 23 March 2012 07:07:14AM 9 points [-]

Interesting question. I guess proofs of algorithm correctness fail less often because:

  1. It's easier to empirically test algorithms to weed out the incorrect ones, so there are fewer efforts to prove conjectures of correctness that are actually false.
  2. It's easier to formalize what it means for an algorithm to be correct than for a cryptosystem to be secure.

In both respects, proving Friendliness seems even worse than proving security.

Powered by Reddit Powered by Reddit