Eugine_Nier comments on Moral Error and Moral Disagreement - Less Wrong

14 Post author: Eliezer_Yudkowsky 10 August 2008 11:32PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (125)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: Vladimir_Nesov 18 November 2010 10:44:20AM *  3 points [-]

The argument seems to be, if Preference1<Archimedes> is too different from Preference1<cousin_it>, then Preference1 is a bad method of preference-extraction and should be rethought. A good method Preference2 for preference-extraction should have Preference2<Archimedes> much closer to Preference2<cousin_it>. And since Preference1 is inadequate, as demonstrated by this test case, Preference1<cousin_it> is also probably hugely worse for cousin_it than Preference2<Archimedes>, even if Preference2<cousin_it> is better than Preference2<Archimedes>.

We are not that wise in the sense that any moral progress we've achieved, if it's indeed progress (so that on reflection, both past and future would agree that the direction was right) and not arbitrary change, shouldn't be a problem for an AI to repeat, and thus this progress in particular (as opposed to other possible differences) shouldn't contribute to differences in extracted preference.

Comment author: Eugine_Nier 18 November 2010 04:23:37PM *  1 point [-]

The argument seems to be, if Preference1<Archimedes> is too different from Preference1<cousin_it>, then Preference1 is a bad method of preference-extraction and should be rethought. A good method Preference2 for preference-extraction should have Preference2<Archimedes> much closer to Preference2<cousin_it>. And since Preference1 is inadequate, as demonstrated by this test case, Preference1<cousin_it> is also probably hugely worse for cousin_it than Preference2<Archimedes>, even if Preference2<cousin_it> is better than Preference2<Archimedes>.

Of course the above constraint isn't nearly enough to uniquely specify Preference2.