= 27d567bc2d48d9f40e9ccc20ea370b15)
MaoShan comments on Not for the Sake of Pleasure Alone - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (129)
Sensible, maybe, but pointless in my opinion. Once you have C1's approval, then any additional improvements (wouldn't C2 like to see what C3 would be like?) would be from C2's perspective, which naturally would be different from C1's perspective, and turtles all the way down. So it would be deceptive to C1 to present him with C2's results, if any incremental happiness were still possible, C2 would naturally harbor the same wish for improvement which caused C1 to accept it. All it would be doing would be shielding C1's virgin imagination from C5822.
I'm not sure you're talking about the same thing I am, or maybe I'm just not following you? There is only C1 and C2. C2 serves as a grounding that checks to see if what it would pick given the experiences it went through is acceptable to C1's initial state. C1 would not have the "virgin imagination", it would be the one hooked up to the wirehead machine.
Really I was thinking about the "Last Judge" idea from the CEV, which (as I understand it, but it is super vague so maybe I don't) basically somehow has someone peek at the solution given by the CEV and decide whether the outcome is acceptable from the outside.
Aside from my accidental swapping of the terms (C1 as the judge, not C2), I still stand by my (unclear, possibly?) opinion. In the situation you are describing, the "judge" would never allow the agent to change beyond a very small distance that the judge is comfortable with, and additional checks would never be necessary, as it would only be logical that the judge's opinion would be the same every time that an improvement was considered. Whichever of the states that the judge finds acceptable the first time, should become the new base state for the judge. Similarly, in real life, you don't hold your preferences to the same standards that you had when you were five years old. The gradual improvements in cognition usually justify the risks of updating one's values, in my opinion.