You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

KatjaGrace comments on Superintelligence 14: Motivation selection methods - Less Wrong Discussion

5 Post author: KatjaGrace 16 December 2014 02:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (28)

You are viewing a single comment's thread.

Comment author: KatjaGrace 16 December 2014 02:21:40AM 2 points [-]

Might it be unethical to make creatures who want to serve your will?

Comment author: Sebastian_Hagen 16 December 2014 06:21:49PM *  6 points [-]

Intelligent minds always come with built-in drives; there's nothing that in general makes goals chosen by another intelligence worse than those arrived through any other process (e.g. natural selection in the case of humans).

One of the closest corresponding human institutions - slavery - has a very bad reputation, and for good reason: Humans are typically not set up to do this sort of thing, so it tends to make them miserable. Even if you could get around that, there's massive moral issues with subjugating an existing intelligent entity that would prefer not to be. Neither of those inherently apply to newly designed entities. Misery is still something that's very much worth avoiding, but that issue is largely orthogonal to how the entity's goals are determined.

Comment author: diegocaleiro 09 February 2015 11:28:20PM 0 points [-]

For a counterargument to your first claim, see the Wisdom of Nature paper by Bostrom and Sandberg (2009 I think).

Comment author: William_S 16 December 2014 07:34:17PM 5 points [-]

If you argue that it would be unethical to make creatures who want to serve your will, would it not be worse to create a creature that does not want to serve your will and use capability control methods to force it to carry out your will anyways?

Comment author: Strilanc 16 December 2014 07:36:32PM 4 points [-]

Possible analogy: Was molding the evolutionary path of wolves, so they turned into dogs that serve us, unethical? Should we stop?

Comment author: Jiro 16 December 2014 08:05:03PM 1 point [-]

I think if it's fair to eat animals, it's far to make them into creatures that serve us.

Comment author: Lumifer 16 December 2014 03:04:24AM 4 points [-]

Might it be unethical to make creatures who want to serve your will?

"You become responsible forever for what you’ve tamed." -- Antoine de Saint-Exupéry