Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

George_Weinberg2 comments on Terminal Values and Instrumental Values - Less Wrong

54 Post author: Eliezer_Yudkowsky 15 November 2007 07:56AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (40)

Sort By: Old

You are viewing a single comment's thread.

Comment author: George_Weinberg2 15 November 2007 08:09:12PM 1 point [-]

I have a question about this picture.

Imagine you have something like a chess playing program. It's got some sort of basic position evaluation function, then uses some sort of look ahead to assign values to the instrumental nodes based on the terminal nodes you anticipate along the path. But unless the game actually ends at the terminal node, it's only "terminal" in the sense that that's where you choose to stop calculating. There's nothing really special about them.

Human beings are different from the chess program in that for us the game never ends, there are no "true" terminal nodes. As you point out, we care what happens after we are dead. So wouldn't it be true that in a sense there's nothing but instrumental values, that a "terminal value" just means that a point at which we've chosen to stop calculating, rather than saying something about the situation itself?