Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Wei_Dai comments on How likely the AI that knows it's evil? Or: is a human-level understanding of human wants enough? - Less Wrong

2 Post author: ChrisHallquist 21 May 2012 05:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (29)

You are viewing a single comment's thread.

Comment author: Wei_Dai 21 May 2012 08:59:54PM 8 points [-]

Personally, I'm inclined to think the bigger risk is an AI with the wrong mix of abilities: say, superhuman abilities in defeating computer security, designing technology, and war planning, but sub-human abilities when it comes understanding what humans want.

That seems likely, don't you think, given that evolution must have optimized us more heavily in the "understanding what humans want" department than the others areas, and understanding other humans is also easier for us since we all share the basic cognitive architecture and can understand others by "putting ourselves in their shoes" (i.e., by imagining what we'd mean by some sentence if we were in their position).