MaoShan comments on Not for the Sake of Pleasure Alone - Less Wrong

36 Post author: lukeprog 11 June 2011 11:21PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (129)

You are viewing a single comment's thread. Show more comments above.

Comment author: MaoShan 14 June 2011 02:54:48AM 0 points [-]

Aside from my accidental swapping of the terms (C1 as the judge, not C2), I still stand by my (unclear, possibly?) opinion. In the situation you are describing, the "judge" would never allow the agent to change beyond a very small distance that the judge is comfortable with, and additional checks would never be necessary, as it would only be logical that the judge's opinion would be the same every time that an improvement was considered. Whichever of the states that the judge finds acceptable the first time, should become the new base state for the judge. Similarly, in real life, you don't hold your preferences to the same standards that you had when you were five years old. The gradual improvements in cognition usually justify the risks of updating one's values, in my opinion.