I find it hard to picture a method of learning what humans value that does not produce information about what they disvalue in equal supply; value is for the most part a relative measure rather than an absolute. (e.g. to determine whether I value eating a cheeseburger it is necessary to compare the state of eating-a-cheeseburger to the state of not-eating-a-cheeseburger, to assess whether I value not-being-in-pain you must compare it to being-in-pain). Is the suggested path, taking this principle into account, that the learner does not produce this information? Some other method, like being forbidden from storing that information? Or is it still an open problem?
I find it hard to picture a method of learning what humans value that does not produce information about what they disvalue in equal supply; value is for the most part a relative measure rather than an absolute. (e.g. to determine whether I value eating a cheeseburger it is necessary to compare the state of eating-a-cheeseburger to the state of not-eating-a-cheeseburger, to assess whether I value not-being-in-pain you must compare it to being-in-pain). Is the suggested path, taking this principle into account, that the learner does not produce this information? Some other method, like being forbidden from storing that information? Or is it still an open problem?