= 27d567bc2d48d9f40e9ccc20ea370b15)
Gunnar_Zarncke comments on Natural selection defeats the orthogonality thesis - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (71)
It's kind of interesting that humans generally don't guard themselves against value drift. Even though any sufficiently intelligent agent clearly would. One of those fundamental divides higher up on the intelligence scale than us, divides that seem binary rather than linear in nature. I wonder if there are any more of those. Apart from (a lack of) susceptibility to the usual biases.
I don't think that 'any' sufficiently intelligent agent 'clearly' would. It requires at least a solution to the cartesianism problem which is currently unsolved and not every self-optimizing process neccessarily solves this.
It's just point 3 from Omohundro's The Basic AI drives paper. Didn't think that's controversial around here. I don't think the Cartesian problem is meant to apply to all power levels (since even plain old humans don't drop anvils on their heads, too often), so the 'sufficiently' ought to cover that objection.
But they do and the reason they mostly don't is found in natural selection and not some inevitable convergence of intelligence.
Any AI that doesn't will have its values drift until they drift to something that guards against value drift.
If that is both abstractly possible and compatible with adaptation. If survival requires constant adaptation, which seems likely, value stability - at least the stability of a precise and concrete set of values - may not be compatible with survival.
Maybe. But in that case the drift implies a selection mechanism - and in the absence of some goal in that direction natural selection applies. Those AI that don't stabilize mutate or stop.
Actually not quite. Until they drift into the core value of existence. Then natural selection will maintain that value, as the AIs that are best at existing will be the ones that exist.
Of course the ones that are best at existing will continue to exist, but I think it is misleading to picture them as a occupying a precise corner of valuespace. Suicidal values are more precise and concrete.