John_Maxwell_IV comments on Formalizing Value Extrapolation - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (48)
(I'm going based on this informal explanation you gave, now.)
It seems like a system such as you describe could exhibit chaotic behavior. Since the person is going to have to create an environment for themselves from scratch, initial decisions about what their environment should be like could impact subsequent decisions, etc. (Also, depending on the level of detail that the person has to specify, reversibility of decisions, etc. maybe the task of creating an environment for oneself would change their character substantially, e.g. like tripping on an untested psychedelic drug.)
Of course, the utility function produced could be "good enough".