= 27d567bc2d48d9f40e9ccc20ea370b15)
Will_Newsome comments on Less Wrong: Open Thread, September 2010 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (610)
It'd be rather easy to twist my words here, but in the case of extrapolated volition it's not like one person gaining the power over the future of mankind is a dystopia or anything.
Let's posit a world where Marcello goes rogue and compiles Marcello-extrapolating AGI. (For anyone who doesn't know Marcello, he's a super awesome guy.) I bet that the resultant universe wouldn't be horrible. Extrapolated-Marcello probably cares about the rest of humanity about as much as extrapolated-humanity does. As humans get smarter and wiser it seems they have a greater appreciation for tolerance, diversity, and all those other lovey-dovey liberal values we implicitly imagine to be the results of CEV. It is unlikely that the natural evolution of 'moral progress' as we understand it will lead to Marcello's extrapolated volition suddenly reversing the the trend and deciding that all other human beings on Earth are essentially worthless and deserve to be stripped to atoms to be turned into a giant Marcello-Pleasure-Simulating Computer. (And even if it did, I believe humans in general are probabilistically similar enough to Marcello that they would count this as positive sum if they knew more and thought faster; but that's a more intricate philosophical argument that I'd rather not defend here.) There are some good arguments to be made for the psychic diversity of mankind, but I doubt the degree of that diversity is enough to tip the scales of utility from positive to negative. Not when diversity is something we seem to have come to appreciate more over time.
This intuition is the result of probably-flawed but at least causal and tractable reasoning about trends in moral progress and the complexity and diversity of human goal structures. It seems that too often when people guess what the results of extrapolated volition will be they use it as a chance to profess and cheer, not carefully predict.
(This isn't much of a response to anything you wrote, katydee; apologies for that. Didn't know where to put it.)
That "best self" thing makes me nervous. People have a wide range of what they think they ought to be, and some of those dreams are ill-conceived.
The scariest kind of dream, perhaps, is exemplified by someone with merely human intelligence who wants to hastily rewrite their own values to conform to their favorite ideology. We'd want an implementation of CEV to recognize this as a bad step in extrapolation. The question is, how do we define what is a "bad step"?