You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Viliam_Bur comments on Why didn't people (apparently?) understand the metaethics sequence? - Less Wrong Discussion

12 Post author: ChrisHallquist 29 October 2013 11:04PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (229)

You are viewing a single comment's thread. Show more comments above.

Comment author: Viliam_Bur 31 October 2013 06:07:33PM *  1 point [-]

If I gave you a source code of a Friendly AI, all you'd have to do would be to run the code.

If I told you to do whatever human CEV is, you'd have to find and interrogate some humans.

The difference is that by analysing the code of the Friendly AI you could probably learn some facts about humans, while by learning about prime numbers you don't learn about the pebblesorters. But that's a consequence of humans caring about humans, and pebblesorters not caring about pebblesorters. Our values are more complex than prime numbers and include caring about ourselves... which is probably likely to happen to a species created by evolution.