You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

wedrifid comments on Arguing Orthogonality, published form - Less Wrong Discussion

10 Post author: Stuart_Armstrong 18 March 2013 04:19PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (10)

You are viewing a single comment's thread.

Comment author: wedrifid 18 March 2013 08:25:42PM 1 point [-]

A motivation might simply be a threat: some truthful powerful being saying “Design an algorithm with goal G. If you succeed, I will give you great goods; if you fail, I will destroy you all. The algorithm will never be used in practice, so there are no moral objections to it being designed.”

Note that this isn't simply a threat. It is a threat combined with an offer. Depending on the nature of the agents in question this can make rather a large difference.