Vladimir_Nesov comments on The Magnitude of His Own Folly - Less Wrong

26 Post author: Eliezer_Yudkowsky 30 September 2008 11:31AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (127)

Sort By: Old

You are viewing a single comment's thread.

Comment author: pdf23ds 01 October 2008 04:03:00PM 2 points [-]

AGI that's complicit with the phasing out of humanity [...] would seem to be at odds with its coded imperative to remain "friendly."

With the CEV definition of Friendliness, it would be Friendly iff that's what humans wanted (in the CEV technical sense). My vision includes that being what humans will want--if I'm wrong about that, a CEV-designed AI wouldn't take us in that direction.

I think the problem of whether what would result would really be the descendants of humanity is directly analogous to the problem of personal identity--if the average atom in the human body has a half-life (of remaining in the body) of two weeks, how can we say we're the same person over time? Evolving patterns. I don't think we really understand either problem too well.