Desrtopa comments on Muehlhauser-Goertzel Dialogue, Part 1 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (161)
A whole lot of the sequences are dedicated to outlining just how reasonably normal people don't act. I would want any Strong AI in charge of our fates to be person-like in that it is aware of what humans want in a way that we would accept, because the alternative to that is probably disaster, but I wouldn't want one to be person-like in that its inductive biases are more like a human's than an ideal Bayesian reasoner's, or that it reasons about moral issues the way humans do intuitively, because our biases are often massively inappropriate, and our moral intuitions incoherent.
Check out this post by Vladimir Nesov: "The problem of choosing Bayesian priors is in general the problem of formalizing preference, it can't be solved completely without considering utility, without formalizing values, and values are very complicated. No simple morality, no simple probability." Of course, having a human prior doesn't necessitate being human-like... Or does it? Duh duh duh.
Today I'd rather say that we don't know if "priors" is a fundamentally meaningful decision-theoretic idea, and so discussing what does or doesn't determine it would be premature.