= 27d567bc2d48d9f40e9ccc20ea370b15)
Vladimir_Nesov comments on Moral Error and Moral Disagreement - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (125)
"Wiser"? What's that mean?
Your comment makes me think that, as of 12 August 2008, you hadn't yet completely given up on your dream of finding a One True Eternal Morality separate from the computation going on in our heads. Have you changed your opinion in the last two years?
The argument seems to be, if Preference1<Archimedes> is too different from Preference1<cousin_it>, then Preference1 is a bad method of preference-extraction and should be rethought. A good method Preference2 for preference-extraction should have Preference2<Archimedes> much closer to Preference2<cousin_it>. And since Preference1 is inadequate, as demonstrated by this test case, Preference1<cousin_it> is also probably hugely worse for cousin_it than Preference2<Archimedes>, even if Preference2<cousin_it> is better than Preference2<Archimedes>.
We are not that wise in the sense that any moral progress we've achieved, if it's indeed progress (so that on reflection, both past and future would agree that the direction was right) and not arbitrary change, shouldn't be a problem for an AI to repeat, and thus this progress in particular (as opposed to other possible differences) shouldn't contribute to differences in extracted preference.
Of course the above constraint isn't nearly enough to uniquely specify Preference2.