Patrick_(orthonormal) comments on You Provably Can't Trust Yourself - Less Wrong

18 Post author: Eliezer_Yudkowsky 19 August 2008 08:35PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (18)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Patrick_(orthonormal) 20 August 2008 01:49:40AM 0 points [-]

Vladimir,

Just to clarify (perhaps unnecessarily): by an attractor I mean a moral framework from which you wouldn't want to self-modify radically in any direction. There do exist many distinct attractors in the space of 'abstracted idealized dynamics', as Eliezer notes for the unfortunate Pebblesorters: they might modify their subgoals, but never approach a morality indifferent to the cardinality of pebble heaps.

Eliezer's claim of moral convergence and the CEV, as I understand it, is that most humans are psychologically constituted so that our moral frameworks lie in the 'basin' of a single attractor; thus the incremental self-modifications of cultural history have an ultimate destination which a powerful AI could deduce.

I suspect, however, that the position is more chaotic than this; that there are distinct avenues of moral progress which will lead us to different attractors. In your terms, since our current right is after all not entirely comprehensive and consistent, we could find that both right1 and right2 are both right extrapolations from right, and that right can't judge unequivocally which one is better.