You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Nisan comments on Brief Question about FAI approaches - Less Wrong Discussion

3 Post author: Dolores1984 19 September 2012 06:05AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (42)

You are viewing a single comment's thread. Show more comments above.

Comment author: Nisan 20 September 2012 08:07:38AM 0 points [-]

I can't formalize what counts as a valid continuation today, let alone in all future societies. So, leave it up to the agents in question.

I think you use the words "valid continuation" to refer to a confused concept. That's why it seems hard to formalize. There is no English sentence that successfully refers to the concept of valid continuation, because it is a confused concept.

If you propose to literally ask models "is this a valid continuation of you?" and simulate them sitting in a room with the future model, then you've got to think about how the models will react to those almost-meaningless words. You might as well ask them "is this a wakalix?".