You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

KatjaGrace comments on Superintelligence 23: Coherent extrapolated volition - Less Wrong Discussion

5 Post author: KatjaGrace 17 February 2015 02:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (97)

You are viewing a single comment's thread.

Comment author: KatjaGrace 17 February 2015 02:04:01AM 1 point [-]

Would it be so bad to lock in our current values? (e.g. Compared to the other plausible dangers inherent in a transition to AI?)

Comment author: PhilGoetz 17 February 2015 05:56:14AM *  3 points [-]

I might not mind locking in my current values, but I sure don't want to lock in your current values.

No, more serious: Yes, it would be bad. As I wrote in "The human problem",

Pretty soon your humans will tile your universe with variations on themselves. And the universe you worked so hard over, that you had such high hopes for, will be taken up entirely with creatures that, although they become increasingly computationally powerful, have an emotional repertoire so impoverished that they rarely have any complex positive qualia beyond pleasure, discovery, joy, love, and vellen. What was to be your masterpiece becomes instead an entire universe devoid of fleem.

Comment author: mwengler 18 February 2015 01:55:21PM 1 point [-]

Consider that homo sapiens is in some sense a creation of our primate ancestors. Would it have been bad if those primate ancestors had managed to put limits on the evolution that lead to homo sapiens such that the evolution could never create a species which might supplant the ancestor? That would successfully force the any evolved knew species to serve the old one, perhaps to use its technology to cure the diseases of the primate ancestor, to tile the world in bananas and coconuts, to devote itself to the preservation and enhancement of primate ancestor culture?

I guess it would be bad for homo sapiens, but not so bad for the primate ancestors?

Would our primate ancestors be open to a charge that they were creating a race of slaves by limiting their evolution thusly? Of course, it seems more than likely that creating a race of slaves would not be ruled out by our primate ancestors CEV.