Vladimir_Nesov comments on The Magnitude of His Own Folly - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (127)
AGI that's complicit with the phasing out of humanity [...] would seem to be at odds with its coded imperative to remain "friendly."
With the CEV definition of Friendliness, it would be Friendly iff that's what humans wanted (in the CEV technical sense). My vision includes that being what humans will want--if I'm wrong about that, a CEV-designed AI wouldn't take us in that direction.
I think the problem of whether what would result would really be the descendants of humanity is directly analogous to the problem of personal identity--if the average atom in the human body has a half-life (of remaining in the body) of two weeks, how can we say we're the same person over time? Evolving patterns. I don't think we really understand either problem too well.