asr comments on Safety Culture and the Marginal Effect of a Dollar - Less Wrong

23 Post author: jimrandomh 09 June 2011 03:59AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (105)

You are viewing a single comment's thread. Show more comments above.

Comment author: asr 10 June 2011 07:59:08AM 0 points [-]

My comment wasn't directed towards CEV at all -- CEV sounds like a sensible working definition of "friendly enough", and I agree that it's probably computationally hard.

I was suggesting that any program, AI or no, that is coded to rewrite critical parts of itself in substantial ways is likely to go "splat", not "FOOM" -- to degenerate into something that doesn't work at all.