Stuart_Armstrong comments on General purpose intelligence: arguing the Orthogonality thesis - Less Wrong

20 Post author: Stuart_Armstrong 15 May 2012 10:23AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (156)
Tags: efficiency rationality intelligence motivation orthogonality ai

Comments (156)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 15 May 2012 12:13:36PM *  9 points [-]

Couple of comments:

  • The section "Bayesian Orthogonality thesis" doesn't seem right, since a Bayesian would think in terms of probabilities rather than possibilities ("could construct superintelligent AIs with more or less any goals"). If you're saying that we should assign a uniform distribution for what AI goals will be realized in the future, that's clearly wrong.
  • I think the typical AI researcher, after reading this paper, will think "sure, it might be possible to build agents with arbitrary goals if one tried, but my approach will probably lead to a benevolent AI". (See here for an example of this.) So I'm not sure why you're putting so much effort into this particular line of argument.
Comment author: Stuart_Armstrong 15 May 2012 12:26:51PM *  4 points [-]

This is the first step (pointed more towards philosophers). Formalise the "we could construct an AI with arbitrary goals", and with that in the background, zoom in on the practical arguments with the AI researchers.

Will restructure the Bayesian section. Some philosophers argue things like "we don't know what moral theories are true, but a rational being would certainly find them"; I want to argue that this is equivalent, from our perspective, with the AI's goals ending up anywhere. What I meant to say is that ignorance of this type is like any other type of ignorance, hence the "Bayesian" terminology.

Comment author: Wei_Dai 15 May 2012 07:28:40PM 4 points [-]

This is the first step (pointed more towards philosophers). Formalise the "we could construct an AI with arbitrary goals", and with that in the background, zoom in on the practical arguments with the AI researchers.

Ok, in that case I would just be wary about people being tempted to cite the paper to AI researchers without having the followup arguments in place, who would then think that their debating/discussion partners are attacking a strawman.

Comment author: Stuart_Armstrong 16 May 2012 12:08:00PM 2 points [-]

Hum, good point; I'll try and put in some disclaimer, emphasising that this is a partial result...

Comment author: Wei_Dai 16 May 2012 12:25:34PM 1 point [-]

Thanks. To go back to my original point a bit, how useful is it to debate philosophers about this? (When debating AI researchers, given that they probably have a limited appetite for reading papers arguing that what they're doing is dangerous, it seems like it would be better to skip this paper and give the practical arguments directly.)

Comment author: Stuart_Armstrong 16 May 2012 12:36:56PM 3 points [-]

Maybe I've spent too much time around philosophers - but there are some AI designers who seem to spout weak arguments like that, and this paper can't hurt. When we get a round to writing a proper justification for AI researchers, having this paper to refer back to avoids going over the same points again.

Plus, it's a lot easier to write this paper first, and was good practice.

Powered by Reddit Powered by Reddit