You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

jacob_cannell comments on Muehlhauser-Wang Dialogue - Less Wrong Discussion

24 Post author: lukeprog 22 April 2012 10:40PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (284)
Tags: agi

Comments (284)

You are viewing a single comment's thread. Show more comments above.

Comment author: Jack 23 April 2012 08:55:24AM *  48 points [-]

I agree. Friendly AI may be incoherent and impossible. In fact, it looks impossible right now. But that’s often how problems look right before we make a few key insights that make things clearer, and show us (e.g.) how we were asking a wrong question in the first place. The reason I advocate Friendly AI research (among other things) is because it may be the only way to secure a desirable future for humanity, (see “Complex Value Systems are Required to Realize Valuable Futures.”) even if it looks impossible. That is why Yudkowsky once proclaimed: “Shut Up and Do the Impossible!” When we don’t know how to make progress on a difficult problem, sometimes we need to hack away at the edges.

Just a suggestion for future dialogs: The amount of Less Wrong jargon, links to Less Wrong posts explaining that jargon, and the Yudkowsky "proclamation" in this paragraph is all a bit squicky, alienating and potentially condescending. And I think they muddle the point you're making.

Anyway, biting Pei's bullet for a moment, if building an AI isn't safe, if it's, like Pei thinks, similar to educating a child (except, presumably, with a few orders of magnitude more uncertainty about the outcome) that sounds like a really bad thing to be trying to do. He writes :

I don’t think a good education theory can be “proved” in advance, pure theoretically. Rather, we’ll learn most of it by interacting with baby AGIs, just like how many of us learn how to educate children.

There's a very good chance he's right. But we're terrible at educating children. Children routinely grow up to be awful people. And this one lacks the predictable, well-defined drives and physical limits that let us predict how most humans will eventually act (pro-social, in fear of authority). It sounds deeply irresponsible, albeit, not of immediate concern. Pei's argument is a grand rebuttal of the proposal that humanity spend more time on AI safety (why fund something that isn't possible?) but no argument at all against the second part of the proposal-- defund AI capabilities research.

Comment author: jacob_cannell 16 June 2012 07:49:44PM *  0 points [-]

The amount of Less Wrong jargon, links to Less Wrong posts explaining that jargon, and the Yudkowsky "proclamation" in this paragraph is all a bit squicky, alienating and potentially condescending.

Yes. Well said. The deeper issue though is the underlying causes of said squicky, alienating paragraphs. Surface recognition of potentially condescending paragraphs is probably insufficient.

Anyway, biting Pei's bullet for a moment, if building an AI isn't safe, if it's, like Pei thinks, similar to educating a child (except, presumably, with a few orders of magnitude more uncertainty about the outcome) that sounds like a really bad thing to be trying to do.

Its unclear that Pei would agree with your presumption that educating an AGI will entail "a few orders of magnitude more uncertainty about the outcome". We can control every aspect of an AGI's development and education to a degree unimaginable in raising human children. Examples: We can directly monitor their thoughts. We can branch successful designs. And perhaps most importantly, we can raise them in a highly controlled virtual environment. All of this suggests we can vastly decrease the variance in outcome compared to our current haphazard approach of creating human minds.

But we're terrible at educating children.

Compared to what? Compared to an ideal education? Your point thus illustrates the room for improvement in educating AGI.

Children routinely grow up to be awful people.

Routinely? Nevertheless, this only shows the scope and potential for improvement. To simplify: if we can make AGI more intelligent, we can also make it less awful.

And this one lacks the predictable, well-defined drives and physical limits that let us predict how most humans will eventually act

An unfounded assumption. To the extent that humans have these "predictable, well-defined drives and physical limits" we can also endow AGI's with these qualities.

Pei's argument is a grand rebuttal of the proposal that humanity spend more time on AI safety (why fund something that isn't possible?) but no argument at all against the second part of the proposal-- defund AI capabilities research.

Which doesn't really require much of an argument against. Who is going to defund AI capabilities research such that this would actually prevent global progress?

Powered by Reddit Powered by Reddit