nshepperd comments on The Meaning of Right - Less Wrong

30 Post author: Eliezer_Yudkowsky 29 July 2008 01:28AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (147)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: nshepperd 28 March 2011 03:24:05AM 0 points [-]

He means a chain of justifications, where each value is only instrumental to the next thing it causes to happen. Not "X is good because I have terminal values XYZ" but "A is good because it will cause B which is good because it will cause C..." which is clearly a silly idea (and completely indeterminate as a description of terminal values).

This is different to what you're saying. Yes, every action has more or less infinite consequences into the far future, so to calculate the expected utility of that action you have to sum the (expected) utility function over all time from now to infinity. Doing this you might find "action A has lots of utility, it's good, because I predict (via this chain of causality) good consequences of X utility at Y probability at time T, and also utility X' probability Y' at time T', and also..." where the utilities are determined by the utility function which encodes your terminal values.

Or, returning to the language of should-ness chains, every action has a more or less infinite chain of consequences, but that doesn't make should-ness an infinite chain. You get the should-ness of an action by adding up (a lot of) finite chains of the form "Action A causes B which causes C which causes D which is good, so A is good." Every chain has a finite length, but there's no limit on how long they can be.