Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

J_Thomas comments on Terminal Values and Instrumental Values - Less Wrong

54 Post author: Eliezer_Yudkowsky 15 November 2007 07:56AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (40)

Sort By: Old

You are viewing a single comment's thread.

Comment author: J_Thomas 15 November 2007 11:20:15AM 4 points [-]

Douglas, in *principle* you ought to consider the entire state of the future universe when you set a terminal value. "I want my sister not to be killed in the next few weeks by flesh-eating bacteria" is a vague goal. "My sister not being killed by flesh-eating bacteria because the world fell into a black hole and tidal effects killed her" is not an adequate alternative.

In practice we set terminal values as if they're independent of everything else. I assume that giving my sister penicillin will not have any side effects I haven't considered. As far as I know she isn't allergic to penicillin. If it will bankrupt me then that's something I will consider. I assume the drug company is not sending its profits to support al qaeda unless somebody comes out and claims it is and the mass media take the claim seriously. I assume the drug company won't use my money to lobby for things I'd disapprove of. I completely ignore the fact that my sister's kidneys will remove the penicillin and she'll repeatedly dose her toilet with a dilute penicillin solution that will encourage the spread of penicillin-resistant bacteria. If I did think about that I might want her to save her urine so it could be treated to destroy the penicillin before it's thrown away.

In practice people think about what they want, and they think about important side effects they have learned to consider, and that's all. If we actually had a holistic view of things we would be very different people.