You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Desrtopa comments on Muehlhauser-Goertzel Dialogue, Part 1 - Less Wrong Discussion

28 Post author: lukeprog 16 March 2012 05:12PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (161)

You are viewing a single comment's thread. Show more comments above.

Comment author: Desrtopa 17 March 2012 02:43:39PM 4 points [-]

Less Wrong memes think that person AIs won't be sufficiently person-like but they sort of tend to assume that conclusion rather than argue for it, which causes memes that aren't familiar with Less Wrong memes to wonder why Less Wrong memes are so incredibly confident that all AIs will necessarily act like autistic OCD people without any possibility at all of acting like normal reasonable people.

A whole lot of the sequences are dedicated to outlining just how reasonably normal people don't act. I would want any Strong AI in charge of our fates to be person-like in that it is aware of what humans want in a way that we would accept, because the alternative to that is probably disaster, but I wouldn't want one to be person-like in that its inductive biases are more like a human's than an ideal Bayesian reasoner's, or that it reasons about moral issues the way humans do intuitively, because our biases are often massively inappropriate, and our moral intuitions incoherent.

Comment author: Will_Newsome 17 March 2012 11:50:26PM 2 points [-]

inductive biases are more like a human's than an ideal Bayesian reasoner's

Check out this post by Vladimir Nesov: "The problem of choosing Bayesian priors is in general the problem of formalizing preference, it can't be solved completely without considering utility, without formalizing values, and values are very complicated. No simple morality, no simple probability." Of course, having a human prior doesn't necessitate being human-like... Or does it? Duh duh duh.

Comment author: Vladimir_Nesov 18 March 2012 12:02:34AM *  3 points [-]

Today I'd rather say that we don't know if "priors" is a fundamentally meaningful decision-theoretic idea, and so discussing what does or doesn't determine it would be premature.