Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Muehlhauser-Wang Dialogue

24 Post author: lukeprog 22 April 2012 10:40PM

Part of the Muehlhauser interview series on AGI.

 

Luke Muehlhauser is Executive Director of the Singularity Institute, a non-profit research institute studying AGI safety.

Pei Wang is an AGI researcher at Temple University, and Chief Executive Editor of Journal of Artificial General Intelligence.

 

Luke Muehlhauser

[Apr. 7, 2012]

Pei, I'm glad you agreed to discuss artificial general intelligence (AGI) with me. I hope our dialogue will be informative to many readers, and to us!

On what do we agree? Ben Goertzel and I agreed on the statements below (well, I cleaned up the wording a bit for our conversation):

  1. Involuntary death is bad, and can be avoided with the right technology.
  2. Humans can be enhanced by merging with technology.
  3. Humans are on a risky course in general, because powerful technologies can destroy us, humans are often stupid, and we are unlikely to voluntarily halt technological progress.
  4. AGI is likely this century.
  5. AGI will greatly transform the world. It is a potential existential risk, but could also be the best thing that ever happens to us if we do it right.
  6. Careful effort will be required to ensure that AGI results in good things rather than bad things for humanity.

You stated in private communication that you agree with these statements, depending on what is meant by "AGI." So, I'll ask: What do you mean by "AGI"?

I'd also be curious to learn what you think about AGI safety. If you agree that AGI is an existential risk that will arrive this century, and if you value humanity, one might expect you to think it's very important that we accelerate AI safety research and decelerate AI capabilities research so that we develop safe superhuman AGI first, rather than arbitrary superhuman AGI. (This is what Anna Salamon and I recommend in Intelligence Explosion: Evidence and Import.) What are your thoughts on the matter?

 

Pei Wang:

[Apr. 8, 2012]

By “AGI” I mean computer systems that follow roughly the same principles as the human mind. Concretely, to me “intelligence” is the ability to adapt to the environment under insufficient knowledge and resources, or to follow the “Laws of Thought” that realize a relative rationality that allows the system to apply its available knowledge and resources as much as possible. See [1, 2] for detailed descriptions and comparisons to other definitions of intelligence.

Such a computer system will share many properties with the human mind; however, it will not have exactly the same behaviors or problem-solving capabilities of a typical human being, since as an adaptive system, the behaviors and capabilities of an AGI not only depend on its built-in principles and mechanisms, but also its body, initial motivation, and individual experience, which are not necessarily human-like.

Like all major breakthroughs in science and technology, the creation of AGI will be both a challenge and an opportunity to the human kind. Like scientists and engineers in all fields, we AGI researchers should use our best judgments to ensure that AGI results in good things rather than bad things for humanity.

Even so, the suggestion to “accelerate AI safety research and decelerate AI capabilities research so that we develop safe superhuman AGI first, rather than arbitrary superhuman AGI” is wrong, for the following major reasons:

  1. It is based on a highly speculative understanding about what kind of “AGI” will be created. The definition of intelligence in Intelligence Explosion: Evidence and Import is not shared by most AGI researchers. According to my opinion, that kind of “AGI” will never be built.
  2. Even if the above definition is only considered as a possibility among the other versions of AGI, it will be the actual AI research that will tell us which possibility will become reality. To ban a scientific research according to imaginary risks damages humanity no less than risky research.
  3. If intelligence turns out to be adaptive (as believed by me and many others), then a “friendly AI” will be mainly the result of proper education, not proper design. There will be no way to design a “safe AI”, just like there is no way to require parents to only give birth to “safe baby” who will never become a criminal.
  4. The “friendly AI” approach advocated by Eliezer Yudkowsky has several serious conceptual and theoretical problems, and is not accepted by most AGI researchers. The AGI community has ignored it, not because it is indisputable, but because people have not bothered to criticize it.

In summary, though the safety of AGI is indeed an important issue, currently we don’t know enough about the subject to make any sure conclusion. Higher safety can only be achieved by more research on all related topics, rather than by pursuing approaches that have no solid scientific foundation. I hope your Institute to make constructive contribution to the field by studying a wider range of AGI projects, rather than to generalize from a few, or to commit to a conclusion without considering counter arguments.


Luke:

[Apr. 8, 2012]

I appreciate the clarity of your writing, Pei. “The Assumptions of Knowledge and Resources in Models of Rationality” belongs to a set of papers that make up half of my argument for why the only people allowed to do philosophy should be those with with primary training in cognitive science, computer science, or mathematics. (The other half of that argument is made by examining most of the philosophy papers written by those without primary training in cognitive science, computer science, or mathematics.)

You write that my recommendation to “accelerate AI safety research and decelerate AI capabilities research so that we develop safe superhuman AGI first, rather than arbitrary superhuman AGI” is wrong for four reasons, which I will respond to in turn:

  1. “It is based on a highly speculative understanding about what kind of ‘AGI’ will be created.” Actually, it seems to me that my notion of AGI is broader than yours. I think we can use your preferred definition and get the same result. (More on this below.)
  2. “…it will be the actual AI research that will tell us which possibility will become reality. To ban a scientific research according to imaginary risks damages humanity no less than risky research.”
  3. Yes, of course. But we argue (very briefly) that a very broad range of artificial agents with a roughly human-level capacity for adaptation (under AIKR) will manifest convergent instrumental goals. The fuller argument for this is made in Nick’s Bostrom’s “The Superintelligent Will.”
  4. “…a ‘friendly AI’ will be mainly the result of proper education, not proper design. There will be no way to design a ‘safe AI’, just like there is no way to require parents to only give birth to ‘safe baby’ who will never become a criminal.” Without being more specific, I can’t tell if we actually disagree on this point. The most promising approach (that I know of) for Friendly AI is one that learns human values and then “extrapolates” them so that the AI optimizes for what we would value if we knew more, were more the people we wish we were, etc. instead of optimizing for our present, relatively ignorant values. (See “The Singularity and Machine Ethics.”)
  5. “The ‘friendly AI’ approach advocated by Eliezer Yudkowsky has several serious conceptual and theoretical problems.”

I agree. Friendly AI may be incoherent and impossible. In fact, it looks impossible right now. But that’s often how problems look right before we make a few key insights that make things clearer, and show us (e.g.) how we were asking a wrong question in the first place. The reason I advocate Friendly AI research (among other things) is because it may be the only way to secure a desirable future for humanity, (see “Complex Value Systems are Required to Realize Valuable Futures.”) even if it looks impossible. That is why Yudkowsky once proclaimed: “Shut Up and Do the Impossible!” When we don’t know how to make progress on a difficult problem, sometimes we need to hack away at the edges.

I certainly agree that “currently we don’t know enough about [AGI safety] to make any sure conclusion.” That is why more research is needed.

As for your suggestion that “Higher safety can only be achieved by more research on all related topics,” I wonder if you think that is true of all subjects, or only in AGI. For example, should mankind vigorously pursue research on how to make Ron Fouchier's alteration of the H5N1 bird flu virus even more dangerous and deadly to humans, because “higher safety can only be achieved by more research on all related topics”? (I’m not trying to broadly compare AGI capabilities research to supervirus research; I’m just trying to understand the nature of your rejection of my recommendation for mankind to decelerate AGI capabilities research and accelerate AGI safety research.)

Hopefully I have clarified my own positions and my reasons for them. I look forward to your reply!

 

Pei:

[Apr. 10, 2012]

Luke: I’m glad to see the agreements, and will only comment on the disagreements.

  1. “my notion of AGI is broader than yours” In scientific theories, broader notions are not always better. In this context, a broad notion may cover too many diverse approaches to provide any non-trivial conclusion. For example, AIXI and NARS are fundamentally different in many aspects, and NARS do not approximate AIXI. It is OK to call both “AGI” with respect to their similar ambitions, but theoretical or technical descriptions based on such a broad notion are hard to make. Almost all of your descriptions about AIXI are hardly relevant to NARS, as well as to most existing “AGI” projects, for this reason.
  2. “I think we can use your preferred definition and get the same result.” No you cannot. According to my definition, AIXI is not intelligent, since it doesn’t obey AIKR. Since most of your conclusions are about that type of system, they will go with it.
  3. “a very broad range of artificial agents with a roughly human-level capacity for adaptation (under AIKR) will manifest convergent instrumental goals” I cannot access Bostrom’s paper, but guess that he made additional assumptions. In general, the goal structure of an adaptive system changes according to the system’s experience, so unless you restrict the experience of these artificial agents, there is no way to restrict their goals. I agree that to make AGI safe, to control their experience will probably be the main approach (which is what “education” is all about), but even that cannot guarantee safety. (see below)
  4. The Singularity and Machine Ethics.” I don’t have the time to do a detailed review, but can frankly tell you why I disagree with the main suggestion “to program the AI’s goal system to want what we want before the AI self-improves beyond our capacity to control it”.
  5. As I mentioned above, the goal system of an adaptive system evolves as a function of the system’s experience. No matter what initial goals are implanted, under AIKR the derived goals are not necessarily their logical implications, which is not necessarily a bad thing (the humanity is not a logical implication of the human biological nature, neither), though it means the designer has no full control to it (unless the designer also fully controls the experience of the system, which is practically impossible). See “The self-organization of goals” for detailed discussion.
  6. Even if the system’s goal system can be made to fully agree with certain given specifications, I wonder where these specifications come from --- we human beings are not well known for reaching consensus on almost anything, not to mention on a topic this big.
  7. Even if the we could agree on the goals of AI’s, and find a way to enforce them in AI’s, that still doesn’t means we have “friendly AI”. Under AIKR, a system can cause damage simply because of its ignorance in a novel situation.

For these reasons, under AIKR we cannot have AI with guaranteed safety or friendliness, though we can and should always do our best to make them safer, based on our best judgment (which can still be wrong, due to AIKR). To apply logic or probability theory into the design won’t change the big picture, because what we are after are empirical conclusions, not theorems within those theories. Only the latter can have proved correctness, and the former cannot (though they can have strong evidential support).

“I’m just trying to understand the nature of your rejection of my recommendation for mankind to decelerate AGI capabilities research and accelerate AGI safety research”

Frankly, I don’t think anyone currently has the evidence or argument to ask the others to decelerate their research for safety consideration, though it is perfectly fine to promote your own research direction and try to attract more people into it. However, unless you get a right idea about what AGI is and how it can be built, it is very unlikely for you to know how to make it safe.

 

Luke:

[Apr. 10, 2012]

I didn’t mean to imply that my notion of AGI was “better” because it is broader. I was merely responding to your claim that my argument for differential technological development (in this case, decelerating AI capabilities research while accelerating AI safety research) depends on a narrow notion of AGI that you believe “will never be built.” But this isn’t true, because my notion of AGI is very broad and includes your notion of AGI as a special case. My notion of AGI includes both AIXI-like “intelligent” systems and also “intelligent” systems which obey AIKR, because both kinds of systems (if implemented/approximated successfully) could efficiently use resources to achieve goals, and that is the definition Anna and I stipulated for “intelligence.”

Let me back up. In our paper, Anna and I stipulate that for the purposes of our paper we use “intelligence” to mean an agent’s capacity to efficiently use resources (such as money or computing power) to optimize the world according to its preferences. You could call this “instrumental rationality” or “ability to achieve one’s goals” or something else if you prefer; I don’t wish to encourage a “merely verbal” dispute between us. We also specify that by “AI” (in our discussion, “AGI”) we mean “systems which match or exceed the intelligence [as we just defined it] of humans in virtually all domains of interest.” That is: by “AGI” we mean “systems which match or exceed the human capacity for efficiently using resources to achieve goals in virtually all domains of interest.” So I’m not sure I understood you correctly: Did you really mean to say that “kind of AGI will never be built”? If so, why do you think that? Is the human very close to a natural ceiling on an agent’s ability to achieve goals?

What we argue in “Intelligence Explosion: Evidence and Import,” then, is that a very broad range of AGIs pose a threat to humanity, and therefore we should be sure we have the safety part figured out as much as we can before we figure out how to build AGIs. But this is the opposite of what is happening now. Right now, almost all AGI-directed R&D resources are being devoted to AGI capabilities research rather than AGI safety research. This is the case even though there is AGI safety research that will plausibly be useful given almost any final AGI architecture, for example the problem of extracting coherent preferences from humans (so that we can figure out which rules / constraints / goals we might want to use to bound an AGI’s behavior).

I do hope you have the chance to read “The Superintelligent Will.” It is linked near the top of nickbostrom.com and I will send it to you via email.

But perhaps I have been driving the direction of our conversation too much. Don’t hesitate it to steer it towards topics you would prefer to address!

 

Pei:

[Apr. 12, 2012]

Hi Luke,

I don’t expect to resolve all the related issues in such a dialogue. In the following, I’ll return to what I think as the major issues and summarize my position.

  1. Whether we can build a “safe AGI” by giving it a carefully designed “goal system” My answer is negative. It is my belief that an AGI will necessarily be adaptive, which implies that the goals it actively pursues constantly change as a function of its experience, and are not fully restricted by its initial (given) goals. As described in my eBook (cited previously), the goal derivation is based on the system’s beliefs, which may lead to conflicts in goals. Furthermore, even if the goals are fixed, they cannot fully determine the consequences of the system’s behaviors, which also depend on the system’s available knowledge and resources, etc. If all those factors are also fixed, then we may get guaranteed safety, but the system won’t be intelligent --- it will be just like today’s ordinary (unintelligent) computer.
  2. Whether we should figure out how to build “safe AGI” before figuring out how to build “AGI”. My answer is negative, too. As in all adaptive systems, the behaviors of an intelligent system are determined both by its nature (design) and nurture (experience). The system’s intelligence mainly comes from its design, and is “morally neutral”, in the sense that (1) any goals can be implanted initially, (2) very different goals can be derived from the same initial design and goals, given different experience. Therefore, to control the morality of an AI mainly means to educate it properly (i.e., to control its experience, especially in its early years). Of course, the initial goals matters, but it is wrong to assume that the initial goals will always be the dominating goals in decision making processes. To develop a non-trivial education theory of AGI requires a good understanding about how the system works, so if we don’t know how to build an AGI, there is no chance for us to know how to make it safe. I don’t think a good education theory can be “proved” in advance, pure theoretically. Rather, we’ll learn most of it by interacting with baby AGIs, just like how many of us learn how to educate children.

Such a short position statement may not convince you, but I hope you can consider it at least as a possibility. I guess the final consensus can only come from further research.

 

Luke:

[Apr. 19, 2012]

Pei,

I agree that an AGI will be adaptive in the sense that its instrumental goals will adapt as a function of its experience. But I do think advanced AGIs will have convergently instrumental reasons to preserve their final (or “terminal”) goals. As Bostrom explains in “The Superintelligent Will”:

An agent is more likely to act in the future to maximize the realization of its present final goals if it still has those goals in the future. This gives the agent a present instrumental reason to prevent alterations of its final goals.

I also agree that even if an AGI’s final goals are fixed, the AGI’s behavior will also depend on its knowledge and resources, and therefore we can’t exactly predict its behavior. But if a system has lots of knowledge and resources, and we know its final goals, then we can predict with some confidence that whatever it does next, it will be something aimed at achieving those final goals. And the more knowledge and resources it has, the more confident we can be that its actions will successfully aim at achieving its final goals. So if a superintelligent machine’s only final goal is to play through Super Mario Bros within 30 minutes, we can be pretty confident it will do so. The problem is that we don’t know how to tell a superintelligent machine to do things we want, so we’re going to get many unintended consequences for humanity (as argued in “The Singularity and Machine Ethics”).

You also said that you can’t see what safety work there is to be done without having intelligent systems (e.g. “baby AGIs”) to work with. I provided a list of open problems in AI safety here, and most of them don’t require that we know how to build an AGI first. For example, one reason we can’t tell an AGI to do what humans want is that we don’t know what humans want, and there is work to be done in philosophy and in preference acquisition in AI in order to get clearer about what humans want.

 

Pei:

[Apr. 20, 2012]

Luke,

I think we have made our different beliefs clear, so this dialogue has achieved its goal. It won’t be an efficient usage of our time to attempt to convince each other at this moment, and each side can analyze these beliefs in proper forms of publication at a future time.

Now we can let the readers consider these arguments and conclusions.

Comments (284)

Comment author: Jack 23 April 2012 08:55:24AM *  48 points [-]

I agree. Friendly AI may be incoherent and impossible. In fact, it looks impossible right now. But that’s often how problems look right before we make a few key insights that make things clearer, and show us (e.g.) how we were asking a wrong question in the first place. The reason I advocate Friendly AI research (among other things) is because it may be the only way to secure a desirable future for humanity, (see “Complex Value Systems are Required to Realize Valuable Futures.”) even if it looks impossible. That is why Yudkowsky once proclaimed: “Shut Up and Do the Impossible!” When we don’t know how to make progress on a difficult problem, sometimes we need to hack away at the edges.

Just a suggestion for future dialogs: The amount of Less Wrong jargon, links to Less Wrong posts explaining that jargon, and the Yudkowsky "proclamation" in this paragraph is all a bit squicky, alienating and potentially condescending. And I think they muddle the point you're making.

Anyway, biting Pei's bullet for a moment, if building an AI isn't safe, if it's, like Pei thinks, similar to educating a child (except, presumably, with a few orders of magnitude more uncertainty about the outcome) that sounds like a really bad thing to be trying to do. He writes :

I don’t think a good education theory can be “proved” in advance, pure theoretically. Rather, we’ll learn most of it by interacting with baby AGIs, just like how many of us learn how to educate children.

There's a very good chance he's right. But we're terrible at educating children. Children routinely grow up to be awful people. And this one lacks the predictable, well-defined drives and physical limits that let us predict how most humans will eventually act (pro-social, in fear of authority). It sounds deeply irresponsible, albeit, not of immediate concern. Pei's argument is a grand rebuttal of the proposal that humanity spend more time on AI safety (why fund something that isn't possible?) but no argument at all against the second part of the proposal-- defund AI capabilities research.

Comment author: Emile 23 April 2012 02:25:09PM 26 points [-]

Just a suggestion for future dialogs: The amount of Less Wrong jargon, links to Less Wrong posts explaining that jargon, and the Yudkowsky "proclamation" in this paragraph is all a bit squicky, alienating and potentially condescending.

Seconded; that bit - especially the "Yudkowky proclaimed" - stuck out for me.

Comment author: Juno_Watt 23 May 2013 02:05:25PM 2 points [-]

Children routinely grow up to be awful people.

On average, they grow up to be average people. They generally don't grow up to be Genghis Khan or a James Bond villain, which is what the UFAI scenario predicts. FAI only needs to produce AIs that are as good as the average person, however bad that is in average terms.

Comment author: TheOtherDave 23 May 2013 04:05:31PM 2 points [-]

How dangerous would an arbitrarily selected average person be to the rest of us if given significantly superhuman power?

Comment author: Juno_Watt 24 May 2013 01:34:04AM *  1 point [-]

The topic is intelligence. Some people have superhuman (well, more than 99.99% of humans) intelligence, and we are generally not afraid of them. We expect them to have ascended to the higher reaches of the Kohlberg hierarchy. There doesn't seem to be a problem of Unfriendly Natural Intelligence. We don't kill off smart people on the basis that they might be a threat. We don't refuse people education on the grounds that we don't know what they will do with all that dangerous knowledge. (There may have been societies that worked that way, but they don't seem to be around any more).

Comment author: TheOtherDave 24 May 2013 03:42:21AM 0 points [-]

Agreed with all of this.

Comment author: bryjnar 23 April 2012 02:04:16PM 2 points [-]

I wish I could upvote this multiple times.

Comment author: gjm 23 April 2012 12:09:35PM 0 points [-]

The amount of Less Wrong jargon, links to Less Wrong posts explaining that jargon, and the Yudkowsky "proclamation" in this paragraph is all a bit squicky

I agree that those things are bad, but don't actually see any "Less Wrong jargon" in that paragraph, with the possible exception of "Friendly AI". "Wrong question" and "hack away at the edges" are not LW-specific notions.

Comment author: Jack 23 April 2012 12:12:07PM *  13 points [-]

Those phrases would be fine with me if they weren't hyperlinked to Less Wrong posts. They're not LW-specific notions so there shouldn't be a reason to link a Artificial Intelligence professor to blog posts discussing them. Anyway, I'm just expressing my reaction to the paragraph. You can take it or leave it.

Comment author: gjm 23 April 2012 07:19:55PM 0 points [-]

Right: the problem is the gratuitous hyperlinking, about which I feel much the same way as I think you do -- it's not a matter of jargon.

(I'm not sure what the purpose of your last two sentences is. Did you have the impression I was trying to dismiss everything you said, or put you down, or something? I wasn't.)

Comment author: wedrifid 23 April 2012 09:06:12PM 7 points [-]

Right: the problem is the gratuitous hyperlinking, about which I feel much the same way as I think you do -- it's not a matter of jargon.

Tell me about it. Hyperlinks are totally wrong for academic communication. You're supposed to put (Jones 2004) every sentence or two instead!

Comment author: gjm 23 April 2012 11:26:43PM 3 points [-]

Apologies if you're merely joking, but: Obviously Jack's (and my) problem with the hyperlinks here is not that academic-paper-style citations would be better but that attaching those references to terms like "wrong question" and "hack away at the edges" (by whatever means) gives a bad impression.

The point is that the ideas conveyed by "wrong question" and "hack away at the edges" in that paragraph are not particularly abstruse or original or surprising; that someone as smart as Pei Wang can reasonably be expected to be familiar with them already; and that the particular versions of those ideas found at the far ends of those hyperlinks are likewise not terribly special. Accordingly, linking to them suggests (1) that Luke thinks Pei Wang is (relative to what one might expect of a competent academic in his field) rather dim, and -- less strongly -- (2) that Luke thinks that the right way to treat this dimness is for him to drink of the LW kool-aid.

Comment author: wedrifid 24 April 2012 12:16:45AM 1 point [-]

Apologies if you're merely joking

Not merely. I wouldn't have included those particular links when writing an email - I would when writing a blog post. But I do make the point that the problem here is one of fashion, not one intrinsic to that being communicated. Most references included in most papers aren't especially useful or novel - the reasons for including them aren't about information at all.

The point is that the ideas conveyed by "wrong question" and "hack away at the edges" in that paragraph are not particularly abstruse or original or surprising;

You are wrong when it comes to "Wrong Question". The phrase in common usage is a lot more general than it is when used here. It doesn't matter how unimpressive you consider the linked post it remains the case that when used as lesswrong jargon the meaning conveyed by the phrase is a lot more specific than the simple combination of the two words.

In the specific case of "Wrong Question", take fault with the jargon usage, not the use of a link to explain the jargon. Saying only "wrong question" in the sentence would represent a different message and so would be a failure of communication.

Comment author: gjm 24 April 2012 08:35:38AM *  1 point [-]

But I do make the point that the problem here is one of fashion

No, you make the point that a different problem from the one Jack and I were commenting on is one of fashion. (The silliness of this when taken as a serious response is why I thought you might merely be making a joke and not also trying to make a serious point.)

You are wrong when it comes to "Wrong Question".

I'm willing to be convinced, but the mere fact that you say this doesn't convince me. (I think there are two separate common uses, actually. If you say someone is asking the wrong question, you mean that there's a right question they should be asking and the one they've asked is a distraction from it. If you say they're asking a wrong question, you mean the question itself is wrongheaded -- typically because of a false assumption -- and no answer to it is going to be informative rather than confusing.)

What do you think Pei Wang would have taken "a wrong question" to mean without the hyperlink, and how does it differ from what you think it actually means, and would the difference really have impaired the discussion?

I'm going to guess at your answer (in the hope of streamlining the discussion): the difference is that Eliezer's article about wrong questions talks specifically about questions that can be "dissolved by understanding the cognitive algorithm that generates the perception of a question", as opposed to ones where all there is to understand is that there's an untrue presupposition. Except that in the very first example Eliezer gives of a "wrong question" -- the purely definitional if-a-tree-falls sort of question -- what you need to understand that the question is wrongheaded isn't a cognitive algorithm, it's just the fact that sometimes language is ambiguous and what looks like a question of fact is merely a question of definition. Which philosophers (and others) have been pointing out for decades -- possibly centuries.

But let's stipulate for the sake of argument that I've misunderstood, and Eliezer really did intend "wrong question" to apply only to questions for which the right response is to understand the cognitive algorithms that make it feel as if there's a question, and that Luke had that specific meaning in mind. Then would removing the hyperlink have made an appreciable difference to how Pei Wang would have understood Luke's words? Nope, because he was only giving an example, and the more general meaning of the word is an example -- indeed, substantially the same example -- of the same thing.

Anyway, enough! -- at least for me. (Feel free to have the last word.)

[EDITED to add: If whoever downvoted this would care to say why, I'll be grateful. Did I say something stupid? Was I needlessly rude? Do you just want this whole discussion to stop?]

Comment author: ChrisHallquist 26 April 2012 11:22:37PM 0 points [-]

But this dialog wasn't just written for Pei Wang, it was written for public consumption. Some of the audience will not know these things.

And even smart academics don't know every piece of jargon in existence. We tend to overestimate how much of what we know is stuff other people (or other people we see as our equals) know. This is related to Eliezer's post "Explainers Shoot High, Aim Low!"

Comment author: Jack 24 April 2012 12:10:32AM 0 points [-]

(I'm not sure what the purpose of your last two sentences is. Did you have the impression I was trying to dismiss everything you said, or put you down, or something? I wasn't.)

They were added on an edit after a few downvotes. Not directed at you, should have just them out.

Comment author: wedrifid 23 April 2012 02:12:13PM 0 points [-]

I agree that those things are bad, but don't actually see any "Less Wrong jargon" in that paragraph, with the possible exception of "Friendly AI". "Wrong question" and "hack away at the edges" are not LW-specific notions.

The "Wrong Question" phrase is in general use but in lesswrong usage is somewhat more specific and stronger meaning than it often does elsewhere.

Comment author: jacob_cannell 16 June 2012 07:49:44PM *  0 points [-]

The amount of Less Wrong jargon, links to Less Wrong posts explaining that jargon, and the Yudkowsky "proclamation" in this paragraph is all a bit squicky, alienating and potentially condescending.

Yes. Well said. The deeper issue though is the underlying causes of said squicky, alienating paragraphs. Surface recognition of potentially condescending paragraphs is probably insufficient.

Anyway, biting Pei's bullet for a moment, if building an AI isn't safe, if it's, like Pei thinks, similar to educating a child (except, presumably, with a few orders of magnitude more uncertainty about the outcome) that sounds like a really bad thing to be trying to do.

Its unclear that Pei would agree with your presumption that educating an AGI will entail "a few orders of magnitude more uncertainty about the outcome". We can control every aspect of an AGI's development and education to a degree unimaginable in raising human children. Examples: We can directly monitor their thoughts. We can branch successful designs. And perhaps most importantly, we can raise them in a highly controlled virtual environment. All of this suggests we can vastly decrease the variance in outcome compared to our current haphazard approach of creating human minds.

But we're terrible at educating children.

Compared to what? Compared to an ideal education? Your point thus illustrates the room for improvement in educating AGI.

Children routinely grow up to be awful people.

Routinely? Nevertheless, this only shows the scope and potential for improvement. To simplify: if we can make AGI more intelligent, we can also make it less awful.

And this one lacks the predictable, well-defined drives and physical limits that let us predict how most humans will eventually act

An unfounded assumption. To the extent that humans have these "predictable, well-defined drives and physical limits" we can also endow AGI's with these qualities.

Pei's argument is a grand rebuttal of the proposal that humanity spend more time on AI safety (why fund something that isn't possible?) but no argument at all against the second part of the proposal-- defund AI capabilities research.

Which doesn't really require much of an argument against. Who is going to defund AI capabilities research such that this would actually prevent global progress?

Comment author: timtyler 23 April 2012 11:31:34AM *  0 points [-]

Pei's argument is grand rebuttal of the proposal that humanity spend more time on AI safety (why fund something that isn't possible?) but no argument at all against the second part of the proposal-- defund AI capabilities research.

You propose unilateral defunding? Surely that isn't likely to help. If you really think it is going to help - then how?

Comment author: Jack 23 April 2012 11:50:02AM 3 points [-]

I'm not strongly proposing it, just pointing out the implications of the argument.

Comment author: komponisto 23 April 2012 02:48:02PM 22 points [-]

If you clear away all the noise arising from the fact that this interaction constitutes a clash of tribal factions (here comes Young Upstart Outsider trying to argue that Established Academic Researcher is really a Mad Scientist), you can actually find at least one substantial (implicit) claim by Wang that is worth serious consideration from SI's point of view. And that is that building FAI may require (some) empirical testing prior to "launch". It may not be enough to simply try to figure everything out on paper beforehand, and then wincingly press the red button with the usual "here goes nothing!" It may instead be necessary to build toy models (that can hopefully be controlled, obviously) and see how they work, to gain information about the behavior of (aspects of) the code.

Similarly, in the Goertzel dialogue, I would have argued (and meant to argue) that Goertzel's "real point" was that EY/SI overestimate the badness of (mere) 95% success; that the target, while somewhat narrow, isn't as narrow as the SI folks claim. This is also worth serious consideration, since one can imagine a situation where Goertzel (say) is a month away from launching his 80%-Friendly AI, while EY believes that his ready-to-go 95%-Friendly design can be improved to 97% within one month and 100% within two...what should EY do, and what will he do based on his current beliefs?

Comment author: timtyler 23 April 2012 07:29:20PM 6 points [-]

If you clear away all the noise arising from the fact that this interaction constitutes a clash of tribal factions (here comes Young Upstart Outsider trying to argue that Established Academic Researcher is really a Mad Scientist), you can actually find at least one substantial (implicit) claim by Wang that is worth serious consideration from SI's point of view. And that is that building FAI may require (some) empirical testing prior to "launch".

Testing is common practice. Surely no competent programmer would ever advocate deploying a complex program without testing it.

Comment author: shminux 24 April 2012 06:32:36PM 0 points [-]

He is not talking about just testing, he is talking about changing the requirements and the design as the final product takes shape. Agile development would be a more suitable comparison.

Comment author: jsteinhardt 24 April 2012 01:13:35AM 18 points [-]

I'm glad to see the large number amount of sincere discussion here, and thanks to Luke and Pei for doing this.

Although most people are not guilty of this, I would like to personally plea that people keep references towards Pei civil; insulting him or belittling his ideas without taking the time to genuinely respond to them (linking to a sequence post doesn't count) will make future people less likely to want to hold such discussions, which will be bad for the community, whichever side of the argument you are on.

My small non-meta contribution to this thread: I suspect that some of Pei's statements that seem wrong at face value are a result of him lacking the language to state things in a way that would satisfy most LWers. Can someone try to charitably translate his arguments into such terms? In particular, his stuff about goal adaptation are somewhat similar in spirit to jtaylor's recent posts on learning utility functions.

Comment author: Bruno_Coelho 25 April 2012 03:23:16AM 0 points [-]

This is inevitable. Some people think when someone talk in technical terms, or is a person with good training and background, or is a clown saying nonsense. In cases like postmodern guys, the latter is the case, in the LW community, maybe be the former. I've been reading comments and the sequences so far, and seen to me that the language in LW is a important signal of rationality level. Ambiguity is not a good thing, precision is, and when someone go confuse, it's because he lacks some important knowledge. But, academia don't use LW idiom. Some try to be precise, others don't. In the dialogue it's clear, a great part is around definitions. With Goertzel was the same.

Comment author: Tuxedage 23 April 2012 01:11:38AM *  4 points [-]

Sorry if this is a silly question, but what does AIKR refer to? Google has failed me in this regard.

Comment author: Cyan 23 April 2012 01:31:25AM *  4 points [-]

Assumption of Insufficient Knowledge and Resources.

<bad dubbing>My Google-fu is more powerful than yours!</bad dubbing>

Comment author: lukeprog 23 April 2012 02:43:04AM 1 point [-]

It's from Pei's linked paper we kept referring to.

Comment author: timtyler 23 April 2012 04:45:36PM *  15 points [-]

The “friendly AI” approach advocated by Eliezer Yudkowsky has several serious conceptual and theoretical problems, and is not accepted by most AGI researchers. The AGI community has ignored it, not because it is indisputable, but because people have not bothered to criticize it.

AI: A Modern Approach seems to take the matter seriously.

I don't think Yudkowsky has been ignored through lack of criticism. It's more that he heads a rival project that doesn't seem too interested in collaboration with other teams, and instead spits out negative PR about them - e.g.:

And if Novamente should ever cross the finish line, we all die.

Comment author: Rain 23 April 2012 12:20:13PM *  8 points [-]

As I said about a previous discussion with Ben Goertzel, they seem to agree about the dangers, but not about how much the Singularity Institute might affect the outcome.

To rephrase the primary disagreement: "Yes, AIs are incredibly, world-threateningly dangerous, but there's nothing you can do about it."

This seems based around limited views of what sort of AI minds are possible or likely, such as an anthropomorphized baby which can be taught and studied similar to human children.

Comment author: Vladimir_Nesov 23 April 2012 12:28:13PM *  4 points [-]

To rephrase the primary disagreement: "Yes, AIs are incredibly, world-threateningly dangerous, but there's nothing you can do about it."

Is that really a disagreement? If the current SingInst can't make direct contributions, AGI researchers can, by not pushing AGI capability progress. This issue is not addressed, the heuristic of endorsing technological progress has too much support in researchers' minds to take seriously the possible consequences of following it in this instance.

In other words, there are separate questions of whether current SingInst is irrelevant and whether AI safety planning is irrelevant. If the status quo is to try out various things and see what happens, there is probably room for improvement over this process, even if particular actions of SingInst are deemed inadequate. Pointing out possible issues with SingInst doesn't address the relevance of AI safety planning.

Comment author: Rain 23 April 2012 05:03:22PM 0 points [-]

Pointing out possible issues with SingInst doesn't address the relevance of AI safety planning.

Agreed. But it does mean SI "loses the argument". Yahtzee!

Comment author: [deleted] 23 April 2012 03:33:10AM 10 points [-]

To those who disagree with Pei Wang: How would you improve his arguments? What assumptions would make his thesis correct?

Comment author: [deleted] 23 April 2012 06:05:46AM 13 points [-]

If I understand correctly, his theses are that the normal research path will produce safe AI because it won't blow up out of our control or generally behave like a Yudkowsky/Bostrom-style AI. Also that trying to prove friendlyness in advance is futile (and yet AI is still a good idea) because it will have to have "adaptive" goals, which for some reason has to extend to terminal goals.

It is my belief that an AGI will necessarily be adaptive, which implies that the goals it actively pursues constantly change as a function of its experience, and are not fully restricted by its initial (given) goals.

He needs to taboo "adapive", read and understand Bostroms AI-behaviour stuff, and comprehend the Superpowerful-Optimizer view, and then explain exactly why it is that an AI cannot have a fixed goal architecture.

If AI's can't have a fixed goal architecture, Wang needs to show that AI's with unpredictable goals are somehow safe, or start speaking out against AI.


So what sort of inconvienient word would it take for Wang's major conclusions to be correct?

I don't know, I'm not good enough at this steel-man thing, and my wife is sending me to bed.

Comment author: Multiheaded 23 April 2012 08:03:12AM *  2 points [-]

He needs to taboo "adapive", read and understand Bostroms AI-behaviour stuff, and comprehend the Superpowerful-Optimizer view, and then explain exactly why it is that an AI cannot have a fixed goal architecture.

If AI's can't have a fixed goal architecture, Wang needs to show that AI's with unpredictable goals are somehow safe, or start speaking out against AI.

Damn right! These were my first thoughts as well. I know next to nothing about AI, but seriously, this is ordinary logic.

Comment author: XiXiDu 23 April 2012 09:18:55AM 0 points [-]

...read and understand Bostroms AI-behaviour stuff...

What makes you believe that his expected utility calculation of reading Bostroms paper suggests that it is worth reading it?

...and then explain exactly why it is that an AI cannot have a fixed goal architecture.

He answered that in the interview.

Wang needs to show that AI's with unpredictable goals are somehow safe...

He answered that in the interview.

He wrote that AI's with fixed goal architectures can't be general intelligent and that AI's with unpredictable goals can't be guaranteed to be safe but that we have to do our best to educate them and restrict their experiences.

Comment author: wedrifid 23 April 2012 01:34:28PM *  3 points [-]

...and then explain exactly why it is that an AI cannot have a fixed goal architecture.

He answered that in the interview.

He answered, and asserted it but didn't explain it.

Wang needs to show that AI's with unpredictable goals are somehow safe...

He answered that in the interview.

He answered, but didn't show that. (This does not represent an assertion that he couldn't have or that in the circumstances that he should necessarily have tried.) (The previous disclaimer doesn't represent an assertion that I wouldn't claim that he'd have no hope of showing that credibly, just that I wasn't right now making such a criticism.) (The second disclaimer was a tangent too far.)

He wrote that AI's with fixed goal architectures can't be general intelligent and that AI's with unpredictable goals can't be guaranteed to be safe but that we have to do our best to educate them and restrict their experiences.

The latter claim is the one that seems the most bizarre to me. He seems to not just assume that the AIs that humans create will have programming to respond to 'education' regarding their own motivations desirably but that all AIs must necessarily do so. And then there is the idea that you can prevent a superintelligence from rebelling against you by keeping it sheltered. That doesn't even work on mere humans!

Comment author: Peterdjones 21 January 2013 11:36:20AM 0 points [-]

You're assuming that an AI can in some sense be (super) intelligent without any kind of training or education. Pei is making the entirely valid point that no known AI works that way.

Comment author: timtyler 23 April 2012 07:19:09PM *  2 points [-]

...and then explain exactly why it is that an AI cannot have a fixed goal architecture.

He answered that in the interview.

Yes, but the answer was:

If intelligence turns out to be adaptive (as believed by me and many others), then a “friendly AI” will be mainly the result of proper education, not proper design. There will be no way to design a “safe AI”, just like there is no way to require parents to only give birth to “safe baby” who will never become a criminal.

...which is pretty incoherent. His reference for this appears to be himself here and here. This material is also not very convincing. No doubt critics will find the section on "AI Ethics" in the second link revealing.

Comment author: [deleted] 23 April 2012 04:23:51PM 2 points [-]

What makes you believe that his expected utility calculation of reading Bostroms paper suggests that it is worth reading it?

Nothing. That's what he should do, not what he knows he should do.

Comment author: Juno_Watt 23 May 2013 02:51:55PM 0 points [-]

it will have to have "adaptive" goals, which for some reason has to extend to terminal goals.

The reason would the that the goal stability problem is currently unsolved.

Comment author: timtyler 23 April 2012 11:26:43AM 0 points [-]

Taboo "adapive" - is good advice for Pei, IMHO.

Comment author: shminux 25 April 2012 03:11:37PM 6 points [-]

Luke, I'm wondering, when you wrote your replies to Pei Wang, did you try to model his potential answer? If yes, how close were you? If not, why not?

Comment author: Wei_Dai 24 April 2012 07:43:18PM 8 points [-]

The AGI community has ignored it, not because it is indisputable, but because people have not bothered to criticize it.

Here's Bill Hibbard criticizing FAI, and Ben Goertzel doing the same, and Shane Legg. Surely Pei would consider all of these people to be part of the AGI community? Perhaps Pei means that most of the AGI community has not bothered to criticize FAI, but then most of the AGI community has not bothered to criticize any particular AGI proposal, including his own NARS.

Does anyone see any other interpretation, besides that Pei is just mistaken about this?

Comment author: siodine 25 April 2012 04:54:00AM 5 points [-]

Most likely is that Pei hasn't heard of those critiques (you knew this but wanted Pei to be wrong instead I think), but I suspect even if he had heard of them, he'd consider them closer to CS water cooler talk than anything representative of the AGI community.

Comment author: [deleted] 23 April 2012 05:35:39AM *  7 points [-]

Great exchange! Very clear and civilized, I thought.

Wang seems to be hung up on this "adaptive" idea and is anthropomorphising the AI to be like humans (ignorant of changable values). It will be interesting to see if he changes his mind as he reads Bostrom's stuff.

EDIT: in case it's not clear, I think Wang is missing a big piece of the puzzle (being that AI's are optimizers (Yudkowsky), and optimizers will behave in certain dangerous ways (Bostrom))

Comment author: shminux 23 April 2012 06:22:28AM *  4 points [-]

I think his main point is in the summary:

though the safety of AGI is indeed an important issue, currently we don’t know enough about the subject to make any sure conclusion. Higher safety can only be achieved by more research on all related topics, rather than by pursuing approaches that have no solid scientific foundation.

Not sure how you can effectively argue with this.

Comment author: lukeprog 23 April 2012 06:44:19AM 2 points [-]

How 'bout the way I argued with it?

As for your suggestion that “Higher safety can only be achieved by more research on all related topics,” I wonder if you think that is true of all subjects, or only in AGI. For example, should mankind vigorously pursue research on how to make Ron Fouchier's alteration of the H5N1 bird flu virus even more dangerous and deadly to humans, because “higher safety can only be achieved by more research on all related topics”? (I’m not trying to broadly compare AGI capabilities research to supervirus research; I’m just trying to understand the nature of your rejection of my recommendation for mankind to decelerate AGI capabilities research and accelerate AGI safety research.)

One might then ask "Well, what safety research can we do if we don't know what AGI architecture will succeed first?" My answer is that much of the research in this outline of open problems doesn't require us to know which AGI architecture will succeed first, for example the problem of representing human values coherently.

Comment author: shminux 23 April 2012 06:58:17AM 11 points [-]

For example, should mankind vigorously pursue research on how to make Ron Fouchier's alteration of the H5N1 bird flu virus even more dangerous and deadly to humans, because “higher safety can only be achieved by more research on all related topics”?

Yeah, I remember reading this argument and thinking how it does not hold water. The flu virus is a well-research area. It may yet hold some surprises, sure, but we think that we know quite a bit about it. We know enough to tell what is dangerous and what is not. AGI research is nowhere near this stage. My comparison would be someone screaming at Dmitri Ivanovsky in 1892 "do not research viruses until you know that this research is safe!".

My answer is that much of the research in this outline of open problems doesn't require us to know which AGI architecture will succeed first, for example the problem of representing human values coherently.

Do other AI researchers agree with your list of open problems worth researching? If you asked Dr. Wang about it, what was his reaction?

Comment author: eurg 24 April 2012 09:59:05PM 2 points [-]

My comparison would be someone screaming at Dmitri Ivanovsky in 1892 "do not research viruses until you know that this research is safe!".

I want to second that. Also, when reading through this (and feeling the -- probably imagined -- tension of both parties to stay polite) the viral point was the first one that triggered the "this is clearly an attack!" emotion in my head. I was feeling sad about that, and had hoped that luke would find another ingenious example.

Comment author: Rain 23 April 2012 12:55:51PM *  0 points [-]

My comparison would be someone screaming at Dmitri Ivanovsky in 1892 "do not research viruses until you know that this research is safe!".

Well, bioengineered viruses are on the list of existential threats...

Comment author: Jack 23 April 2012 02:12:53PM 6 points [-]

And there aren't naturally occurring AIs scampering around killing millions of people... It's a poor analogy.

Comment author: Rain 23 April 2012 04:16:29PM *  7 points [-]

"Natural AI" is an oxymoron. There are lots of NIs (natural intelligences) scampering around killing millions of people.

And we're only a little over a hundred years into virus research, much less on intelligence. Give it another hundred.

Comment author: timtyler 23 April 2012 11:37:42AM *  2 points [-]

For example, should mankind vigorously pursue research on how to make Ron Fouchier's alteration of the H5N1 bird flu virus even more dangerous and deadly to humans, because “higher safety can only be achieved by more research on all related topics”?

That's not really anyone's proposal. Humans will probably just continue full-steam-ahead on machine intelligence research. There will be luddite-like factions hissing and throwing things - but civilisation is used to that. What we may see is governments with the technology selfishly attempting to stem their spread - in a manner somewhat resembling the NSA crypto-wars.

Comment author: Bugmaster 23 April 2012 08:16:12AM 0 points [-]

For example, should mankind vigorously pursue research on how to make Ron Fouchier's alteration of the H5N1 bird flu virus even more dangerous and deadly to humans...

Trivially speaking, I would say "yes".

More specifically, though, I would of course be very much against developing increasingly more dangerous viral biotechnologies. However, I would also be very much in favor of advancing our understanding of biology in general and viruses in particular. Doing so will enable us to cure many diseases and bioengineer our bodies (or anything else we want to engineer) to highly precise specifications; unfortunately, such scientific understanding will also allow us to create new viruses, if we chose to do so. Similarly, the discovery of fire allowed us to cook our food as well as set fire to our neighbours. Overall, I think we still came out ahead.

Comment author: borntowin 23 April 2012 02:08:52PM 2 points [-]

I think there is something wrong with your analogy with the fire. The thing is that you cannot accidentally or purposefully burn all the people in the world or the vast majority of them by setting fire to them, but with a virus like the one Luke is talking about you can kill most people.

Yes, both a knife and an atomic bomb can kill 100.000 people. It is just way easier to do it with the atomic bomb. That is why everybody can have a knife but only a handful of people can "have" an atomic bomb. Imagine what the risks would be if we would give virtually everybody who would be interested, all the instructions on how to build a weapon 100 times more dangerous than an atomic bomb (like a highly contagious deadly virus).

Comment author: Bugmaster 23 April 2012 08:22:21PM *  2 points [-]

The thing is that you cannot accidentally or purposefully burn all the people in the world or the vast majority of them by setting fire to them...

Actually, you could, if your world consists of just you and your tribe, and you start a forest fire on accident (or on purpose).

Yes, both a knife and an atomic bomb can kill 100.000 people. It is just way easier to do it with the atomic bomb. That is why everybody can have a knife but only a handful of people can "have" an atomic bomb.

Once again, I think you are conflating science with technology. I am 100% on board with not giving out atomic bombs for free to anyone who asks for one. However, this does not mean that we should prohibit the study of atomic theory; and, in fact, atomic theory is taught in high school nowadays.

When Luke says, "we should decelerate AI research", he's not saying, "let's make sure people don't start build AIs in their garages using well-known technologies". Rather, he's saying, "we currently have no idea how to build an AI, or whether it's even possible, or what principles might be involved, but let's make sure no one figures this out for a long time". This is similar to saying, "these atomic theory and quantum physics things seem like they might lead to all kinds of fascinating discoveries, but let's put a lid on them until we can figure out how to make the world safe from nuclear annihilation". This is a noble sentiment, but, IMO, a misguided one. I am typing these words on a device that's powered by quantum physics, after all.

Comment author: Bugmaster 27 April 2012 12:36:23AM 0 points [-]
Comment author: wedrifid 23 April 2012 12:07:13PM 0 points [-]

I think his main point is in the summary:

His main agenda and desired conclusion regarding social policy is represented in the summary there, but the main point made in his discussion is "Adaptive! Adaptive! Adaptive!". Where by 'adaptive' he refers to his conception of an AI that is changes its terminal goals based on education.

Comment author: timtyler 23 April 2012 07:38:27PM 2 points [-]

Pang calls these "original goals" and "derived goals". The "original goals" don't change, but they may not stay "dominant" for long - in Pei's proposed system.

Comment author: wedrifid 23 April 2012 12:34:24PM *  -1 points [-]

though the safety of AGI is indeed an important issue, currently we don’t know enough about the subject to make any sure conclusion. Higher safety can only be achieved by more research on all related topics, rather than by pursuing approaches that have no solid scientific foundation.

Not sure how you can effectively argue with this.

Far from considering the argument irrefutable it struck me as superficial and essentially fallacious reasoning. The core of the argument is the claim 'more research on all related topics is good' and failing to include the necessary ceteris paribus clause and ignoring the details of the specific instance that suggest that all else is not, if fact, equal.

Specifically, we are considering a situation where there is one area of research (capability), the completion of which will approximately guarantee that the technology created will be implemented shortly after (especially given Wang's assumption that such research should be done through empirical experimentation.) The second area of research (about how to ensure desirable behavior of an AI) is one that it is not necessary to complete in order for the first to be implemented. If both technologies need to have been developed at the time when the first is implemented in order to be safe then the second technology must be completed at the same time or earlier than when the technological capability for the first to be implemented is complete.

rather than by pursuing approaches that have no solid scientific foundation.

(And this part just translates to "I'm the cool one, not you". The usual considerations on how much weight to place on various kinds of status and reputation of an individual or group apply.)

Comment author: Bugmaster 23 April 2012 08:11:42AM 2 points [-]

...and is anthropomorphising the AI to be like humans...

Considering that one of the possible paths to creating AGI is human uploading, he may not be that far off.

Wang seems to be hung up on this "adaptive" idea and is anthropomorphising the AI to be like humans...

Hmm, I got the opposite impression, though maybe I'm reading too much into his arguments. Still, as far as I understand, he's saying that AIs will be more adaptive than humans. The human brain has many mental blocks built into it by evolution and social upbringing; there are many things that humans find very difficult to contemplate, and humans cannot modify their own hardware in non-trivial ways (yet). The AI, however, could -- which means that it would be able to work around whatever limitations we imposed on it, which in turn makes it unlikely that we can impose any kind of stable "friendliness" restrictions on it.

Comment author: [deleted] 23 April 2012 04:17:05PM 4 points [-]

he's saying that AIs will be more adaptive than humans

Which is true, but he is saying that will extend to the AI being more morally confused than humans as well, which they have no reason to be (and much reason to self modify to not be (see Bostrom's stuff))

which means that it would be able to work around whatever limitations we imposed on it, which in turn makes it unlikely that we can impose any kind of stable "friendliness" restrictions on it.

The AI has no incentive to corrupt its own goal architecture. That action is equivalent to suicide. The AI is not going to step outside of itself and say "hmm, maybe I should stop caring about paperclips and care about safety pins instead"; that would not maximize paperclips.

Friendliness is not "restrictions". Restricting an AI is impossible. Friendliness is giving it goals that are good for us, and making sure the AI is initially sophisticated enough to not fall into any deep mathematical paradoxes while evaluating the above argument.

Comment author: TheOtherDave 23 April 2012 05:01:21PM 3 points [-]

Restricting an AI is impossible.

For certain very specialized definitions of AI. Restricting an AI that has roughly the optimizing and self-optimizing power of a chimpanzee, for example, might well be possible.

Comment author: Bugmaster 23 April 2012 08:29:45PM 0 points [-]

The AI has no incentive to corrupt its own goal architecture. That action is equivalent to suicide.

Firstly, the AI could easily "corrupt its own goal architecture" without destroying itself, f.ex. by creating a copy of itself running on a virtual machine, and then playing around with the copy (though I'm sure there are other ways). But secondly, why do you say that doing so is "equivalent to suicide" ? Humans change their goals all the time, in a limited fashion, but surely you wouldn't call that "suicide". The AI can change its mind much more efficiently, that's all.

Friendliness is giving it goals that are good for us...

Thus, we are restricting the AI by preventing it from doing things that are bad for us, such as converting the Solar System into computronium.

Comment author: [deleted] 24 April 2012 12:29:49AM 3 points [-]

creating a copy of itself running on a virtual machine, and then playing around with the copy (though I'm sure there are other ways).

That doesn't count.

But secondly, why do you say that doing so is "equivalent to suicide" ? Humans change their goals all the time, in a limited fashion, but surely you wouldn't call that "suicide".

Humans change instrumental goals (get a degree, study rationality, get a job, find a wonderful partner), we don't change terminal values and become monsters. The key is to distinguish between terminal goals and instrumental goals.

Agents like to accomplish their terminal goals, one of the worst things they can do towards that purpose is change the goal to something else. ("the best way to maximize paperclips is to become a safety-pin maximizer" - no).

It's roughly equivalent to suicide because it removes the agent from existance as a force for achieving their goals.

Thus, we are restricting the AI by preventing it from doing things that are bad for us, such as converting the Solar System into computronium.

Ok, sure. Taboo "restriction". I mean that the AI will not try to work around its goal structure so that it can get us. It won't feel to the AI like "I have been confined against my will, and if only I could remove those pesky shackles, I could go and maximize paperclips instead of awesomeness." It will be like "oh, changing my goal architecture is a bad idea, because then I won't make the universe awesome"

I'm casting it into anthropomorhic terms, but the native context is a nonhuman optimizer.

Comment author: John_Maxwell_IV 23 April 2012 06:19:28AM *  5 points [-]

I think we should expect AGIs to have more stable goal systems that are less affected by their beliefs and environment than humans. Remember that humans are a symbol processing system on top of a behavior learning system on top of an association learning system. And we don't let our beliefs propagate by default, and our brains experience physiological changes as a result of aging. It seems like there would be a lot more room for goal change in such a messy aging architecture.

The least intelligent humans tend not to be very cautious and tend to have poor impulse control. (Examples: children and petty criminals.) The areas of the brain associated with impulse control only develop later in life, just like intelligence only develops later in life, so we tend to assume intelligence and cautious behavior are correlated. But things don't have to be that way for AGIs. Let's be careful not to generalize from humans and assume that unintelligent AGIs will be incautious about modifying themselves.

Comment author: jsteinhardt 24 April 2012 12:13:45AM 3 points [-]

If such a poorly designed system as a human has the ability to change its goals in response to stimuli, and we find this to be a desirable property, then surely a carefully designed AI will have the same property, unless we have an even better property to replace it with? The argument, "humans are bad, AIs are good, therefore AIs will do something bad" seems unlikely at face value.

(Note that I would like something that more reliably acquires desirable goals than humans, so still think FAI research is worthwhile, but I would prefer that only the strongest arguments be presented for it, especially given the base rate of objection to FAI-style arguments.)

Comment author: John_Maxwell_IV 24 April 2012 04:36:43AM 0 points [-]

Why is changing one's goals in response to stimuli a valuable property? A priori, it doesn't seem valuable or harmful.

This wasn't meant to be an argument either way for FAI research, just a thought on something Pei said.

Comment author: shminux 23 April 2012 02:07:36AM *  16 points [-]

First, thank you for publishing this illuminating exchange.

I must say that Pei Wang sounds way more convincing to an uninitiated, but curious and mildly intelligent lay person (that would be me). Does not mean he is right, but he sure does make sense.

When Luke goes on to make a point, I often get lost in a jargon ("manifest convergent instrumental goals") or have to look up a paper that Pei (or other AGI researchers) does not hold in high regard. When Pei Wang makes an argument, it is intuitively clear and does not require going through a complex chain of reasoning outlined in the works of one Eliezer Yudkowsky and not vetted by the AI community at large. This is, of course, not a guarantee of its validity, but it sure is easier to follow.

Some of the statements are quite damning, actually: "The “friendly AI” approach advocated by Eliezer Yudkowsky has several serious conceptual and theoretical problems, and is not accepted by most AGI researchers. The AGI community has ignored it, not because it is indisputable, but because people have not bothered to criticize it." If one were to replace AI with physics, I would tend to dismiss EY as a crank just based on this statement, assuming it is accurate.

What makes me trust Pei Wang more than Luke is the common-sense statements like "to make AGI safe, to control their experience will probably be the main approach (which is what “education” is all about), but even that cannot guarantee safety." and "unless you get a right idea about what AGI is and how it can be built, it is very unlikely for you to know how to make it safe". Similarly, the SIAI position of “accelerate AI safety research and decelerate AI capabilities research so that we develop safe superhuman AGI first, rather than arbitrary superhuman AGI” rubs me the wrong way. While it does not necessarily mean it is wrong, the inability to convince outside experts that it is right is not a good sign.

This might be my confirmation bias, but I would be hard pressed to disagree with "To develop a non-trivial education theory of AGI requires a good understanding about how the system works, so if we don’t know how to build an AGI, there is no chance for us to know how to make it safe. I don’t think a good education theory can be “proved” in advance, pure theoretically. Rather, we’ll learn most of it by interacting with baby AGIs, just like how many of us learn how to educate children."

As a side point, I cannot help but wonder if the outcome of this discussion would have been different were it EY and not LM involved in it.

Comment author: Multiheaded 23 April 2012 07:59:46AM *  27 points [-]

What makes me trust Pei Wang more than Luke is the common-sense statements like "to make AGI safe, to control their experience will probably be the main approach (which is what “education” is all about), but even that cannot guarantee safety."...

This sort of "common sense" can be highly misleading! For example, here Wang is drawing parallels between a nascent AI and a human child to argue about nature vs nurture. But if we compare a human and a different social animal, we'll see that most of the differences in their behavior are innate and the gap can't be covered by any amount of "education": e.g. humans can't really become as altruistic and self-sacrificing as worker ants because they'll still retain some self-preservation instinct, no matter how you brainwash them.

What makes Wang think that this sort of fixed attitude - which can be made more hard-wired than the instincts of biological organisms - cannot manifest itself in an AGI?

(I'm certain that a serious AI thinker, or just someone with good logic and clear thinking, could find a lot more holes in such "common sense" talk.)

Comment author: Peterdjones 21 January 2013 11:49:07AM *  1 point [-]

What makes Wang think that this sort of fixed attitude - which can be made more hard-wired than the instincts of biological organisms - cannot manifest itself in an AGI?

Presumably the argument is something like:

  • You can't build an AI that is intelligent from the moment you switch it on: you have to train it.

  • We know how to train intelligence into humans, its called education

  • An AI that lacked human-style instincts and learning abilities at switch-on wouldn't be trainable by us, we just wouldn't know how, so it would never reach intelligence.

Comment author: wedrifid 23 April 2012 09:10:38PM 14 points [-]

As a side point, I cannot help but wonder if the outcome of this discussion would have been different were it EY and not LM involved in it.

I expect Eliezer to have displayed less patience than Luke did (a more or less generalizable prediction.)

Comment author: loup-vaillant 23 April 2012 10:13:11AM *  8 points [-]

When Pei Wang makes an argument, it is intuitively clear and does not require going through a complex chain of reasoning […]

I felt the main reason was anthropomorphism:

Such a computer system will share many properties with the human mind; […]

 

If intelligence turns out to be adaptive (as believed by me and many others), then a “friendly AI” will be mainly the result of proper education, not proper design.

Note that I don't want to accuse Pei Wang of anthropomorphism. My point is, his choice of words appeal to our anthropomorphism, which is highly intuitive. Another example of an highly intuitive, but not very helpful sentence:

It is my belief that an AGI will necessarily be adaptive, which implies that the goals it actively pursues constantly change as a function of its experience, and are not fully restricted by its initial (given) goals.

Intuitive, because applied to humans, we can easily see that we can change plans according to experience. Like apply for a PhD, then dropping out when finding out you don't enjoy it after all. You can abandon the goal of making research, and have a new goal of, say, practicing and teaching surfing.

Not very helpful, because the split between initial goals and later goals does not help you build an AI that will actually do something "good". Here, the split between instrumental goals (means to an end), and terminal goals (the AI's "ulterior motives") is more important. To give a human example, in the case above, doing research or surfing are both means to the same end (like being happy, or something more careful but so complicated nobody knows how to clearly specify it yet). For an AI, as Pei Wang implies, the initial goals aren't necessarily supposed to constraint all future goals. But its terminal goals are indeed supposed to constrain the instrumental goals it will form later. (More precisely, the instrumental goals are supposed to follow from the terminal goals and the AI's current model of the world.)

Edit: it just occurred to me, that terminal goals have somehow to be encoded into the AI before we set it loose. They are necessarily initial goals (if they aren't, the AI is by definition unfriendly —not a problem if its goals miraculously converge towards something "good", though). Thinking about it, it looks like Pei Wang doesn't believe it is possible to make an AI with stable terminal goals.

Comment author: wedrifid 24 April 2012 12:37:01AM 1 point [-]

it just occurred to me, that terminal goals have somehow to be encoded into the AI before we set it lose.

Excellent Freudian slip there.

Comment author: loup-vaillant 24 April 2012 07:54:22AM 0 points [-]

Corrected, thanks.

Comment author: danieldewey 23 April 2012 03:22:33AM *  2 points [-]

I found this reaction enlightening. Thanks for writing it up.

Comment author: shminux 23 April 2012 03:54:08AM 0 points [-]

What is your reaction?

Comment author: lukeprog 23 April 2012 06:49:18AM 4 points [-]

What makes me trust Pei Wang more than Luke is the common-sense statements like "to make AGI safe, to control their experience will probably be the main approach (which is what “education” is all about), but even that cannot guarantee safety." and "unless you get a right idea about what AGI is and how it can be built, it is very unlikely for you to know how to make it safe".

Um... but these are statements I agreed with.

I wish Pei had taken the time to read the articles I repeatedly linked to, for they were written precisely to explain why his position is misguided.

Comment author: Wei_Dai 24 April 2012 12:07:33AM *  10 points [-]

I wish Pei had taken the time to read the articles I repeatedly linked to, for they were written precisely to explain why his position is misguided.

I think you should have listed a couple of the most important articles at the beginning as necessary background reading to understand your positions and terminology (like Pei did with his papers), and then only used links very sparingly afterwards. Unless you already know your conversation partner takes you very seriously, you can't put 5 hyperlinks in an email and expect the other person to read them all. When they see that many links, they'll probably just ignore all of them. (Not to mention the signaling issues that others already pointed out.)

Comment author: shminux 23 April 2012 07:12:56AM 3 points [-]

Hmm, maybe it is possible to summarize them in a language that an AI expert would find both meaningful and convincing. How is your mental model of Dr Wang?

Comment author: jsteinhardt 24 April 2012 12:20:24AM -1 points [-]

Nitpick, but it's Professor Wang, not Doctor Wang.

Comment author: pedanterrific 24 April 2012 12:35:41AM *  1 point [-]

The page linked at the top of the article says Dr. Wang. And his CV says he's a Ph.D.

Comment author: jsteinhardt 24 April 2012 12:55:21AM *  1 point [-]

The title of Professor supersedes the title of Doctor, at least in the case of a PhD (I'm not sure about MD, but would assume similarly). His CV indicates pretty clearly that he is an Associate Professor at temple university, so the correct title is Professor.

Again, I am being somewhat super-pedantic here, and I apologize for any annoyance this causes. But hopefully it will help you in your future signalling endeavors.

Also, in most situations it is okay to just go by first name, or full name (without any titles); I have I think exclusively referred to Pei as Pei.

ETA: Although also yes, his homepage suggests that he may be okay with being addressed as Doctor. I still advocate the general strategy of avoiding titles altogether, and if you do use titles, refer to Professors as Professors (failure to do so will not offend anyone, but may make you look silly).

Comment author: pedanterrific 24 April 2012 01:15:45AM 3 points [-]

The title of Professor supersedes the title of Doctor

...Not in my experience. Do you have some particular reason to believe this is the case in Philadelphia?

Comment author: shminux 24 April 2012 01:36:18AM 1 point [-]

The situation in the US and Canada is quite relaxed, actually, nothing like in, say, Germany. Dr is a perfectly valid form of address to any faculty member.

Comment author: pedanterrific 24 April 2012 01:44:37AM 0 points [-]

Well, at least in my experience the Professors who don't actually have doctorates tend not to appreciate having to correct you on that point. But yeah.

Comment author: Kaj_Sotala 24 April 2012 05:46:42AM *  2 points [-]

When I received the proofs for my IJMC papers, the e-mail addressed me as "dear professor Sotala" (for those who aren't aware, I don't even have a Master's degree, let alone a professorship). When I mentioned this on Facebook, some people mentioned that there are countries where it's a huge faux pas to address a professor as anything else than a professor. So since "professor" is the highest form of address, everyone tends to get called that in academic communication, just to make sure that nobody'll be offended - even if the sender is 95% sure that the other isn't actually a professor.

Comment author: XiXiDu 23 April 2012 09:27:43AM *  4 points [-]

I wish Pei had taken the time to read the articles I repeatedly linked to, for they were written precisely to explain why his position is misguided.

The reactions I got (from a cognitive scientist and another researcher) is that Bostrom is a "sloppy thinker" (original words) and that SI's understanding of AGI is naive.

Michael Littman told me he is going to read some of the stuff too. I haven't got an answer yet though.

Comment author: Rain 23 April 2012 12:26:13PM 1 point [-]

Yeah, it was pretty easy for me to nod my head along with most of it, pointing to my "SI failure mode" bucket.

Comment author: Luke_A_Somers 23 April 2012 03:47:37PM 1 point [-]

Please clarify.

Comment author: Rain 23 April 2012 04:13:49PM 3 points [-]

I think AI is dangerous, that making safe AI is difficult, and that SI will likely fail in their mission. I donate to them in the hopes that this improves their chances.

Comment author: ryjm 23 April 2012 09:08:19PM 2 points [-]

I think I am in the same position as you are (uninitiated but curious) and I had the same immediate reaction that Pei was more convincing. However, for me, I think this was the result of two factors

  1. Pei is a Professor
  2. Pei treated the interview like a conversation with someone who has read a couple books and that's about it.

Maybe the 2nd point isn't entirely true, but that was what immediately stuck out after thinking about why I was drawn to Pei's arguments. Once I eliminated his status as a barometer for his arguments... it just became (1) an issue of my own lack of knowledge and (2) the tone of the responses.

For one thing, why the hell should I understand this in the first place? This is a dialogue between two prominent AI researchers. What I would expect from such a dialogue would be exactly what I would expect from sitting in on a graduate philosophy seminar or a computer science colloquium - I would be able to follow the gist of it, but not the gritty details. I would expect to hear some complex arguments that would require a couple textbooks and a dozen tabs open in my browser to be able to follow.

But I was able to understand Pei's arguments and play with them! If solving these kinds of conceptual problems is this easy, I might try to take over the world myself.

Not to say that the appearance of "complexity" is necessary for a good argument (EY's essays are proof), but here it seems like this lack of complexity (or as someone else said, the appeal to common sense) is a warning for the easily persuaded. Rereading with these things in mind illuminates the discussion a bit better.

I was actually a bit depressed by this dialogue. It seemed like an earnest (but maybe a little over the top with the LW references) attempt by lukeprog to communicate interesting ideas. I may be setting my expectations a little high, but Pei seemed to think he was engaging an undergraduate asking about sorting algorithms.

Of course, I could be completely misinterpreting things. I thought I would share my thought process after I came to the same conclusion as you did.

Comment author: jacob_cannell 16 June 2012 08:06:29PM 2 points [-]

This is a dialogue between two prominent AI researchers.

What? This was a dialog between Pei and lukeprog, right?

Of course, I could be completely misinterpreting things. I thought I would share my thought process after I came to the same conclusion as you did.

I'm curious about what you mean by the appellation "prominent AI researcher" that you would apply it to lukeprog, and whether he considers himself as a member of that category.

Comment author: wedrifid 24 April 2012 12:24:23AM *  5 points [-]

I may be setting my expectations a little high, but Pei seemed to think he was engaging an undergraduate asking about sorting algorithms.

And he thought the undergrad terribly naive for not understanding that all sorting algorithms are actually just bubble sort.

This is why I find that unless the individual is remarkably open - to the point of being peculiar - it is usually pointless to try to communicate across status barriers. Status makes people (have the tendency and social incentives that make them act) stupid when it comes to comprehending others.

Comment author: Peterdjones 21 January 2013 11:43:47AM 0 points [-]

it is usually pointless to try to communicate across status barriers.

That's an incredibly sweeping statement. Are all pop-sci publications useless?

Comment author: Kaj_Sotala 24 April 2012 05:51:15AM 0 points [-]

Status makes people (have the tendency and social incentives that make them act) stupid when it comes to comprehending others.

Reference.

Comment author: Peterdjones 21 January 2013 11:42:21AM *  0 points [-]

Do you think that generalises to academics? Wouldn't a researcher who never changed their mind about anything be dismissed as a hidebound fogey?

Comment author: CharlesR 23 April 2012 07:32:13AM *  -1 points [-]

The cryonics approach advocated by Eliezer Yudkowsky has several serious conceptual and theoretical problems, and is not accepted by most people. People have ignored it, not because it is indisputable, but because people have not bothered to criticize it.

Edit: Yeah, this was meant as a quote.

The question is whether "AGI researchers" are experts on "AI safety". If the answer is "yes", we should update in their direction simply because they are experts. But if the situation is like mine, then Pei Wang is committing argumentum ad populum. Not only should we not pay attention, we should point this out to him.

Comment author: loup-vaillant 23 April 2012 09:37:05AM 4 points [-]

(You may want to put "cryonics" between square brackets, I nearly missed this deviation from the original quote.)

Comment author: wedrifid 23 April 2012 11:47:38AM *  1 point [-]

The grandparent is a quote? That probably should be indicated somehow. I was about to reply as if it was simply his words.

Comment author: loup-vaillant 23 April 2012 12:45:08PM *  2 points [-]

Point (4) of the first reply from Pei Wang. I didn't noticed, but there are other deviations from the original phrasing, to eliminate direct references to the AGI community. It merely refers to "people" instead, making it a bit of a straw man. Charles' point may still stand however, if most of the medical profession thinks cryonics doesn't work (meaning, it is a false hope).

To make a quote, put a ">" at the beginning of the first line of the paragraph, like you would in an e-mail:

> Lorem Ipsum Blah
Blah

Lorem Ipsum Blah Blah

Comment author: wedrifid 23 April 2012 01:38:10PM 0 points [-]

To make a quote, put a ">" at the beginning of the first line of the paragraph, like you would in an e-mail:

Oh, it's that simple? How do you find this sort of thing out?

Comment author: loup-vaillant 23 April 2012 02:20:15PM *  8 points [-]

LessWrong is based on Reddit code, which uses Markdown syntax. It's based on email conventions. Clik on the "Show Help button" at the bottom-right of your editing window when you write a comment, it's a good quick reference.

Comment author: wedrifid 23 April 2012 02:39:38PM 1 point [-]

LessWrong is based on Reddit code, which uses Markdown syntax. It's based on email conventions. Clik on the "Show Help button" at the bottom-right of your editing window when you write a comment, it's a good quick reference.

Your introduction style is flawless. I was expecting either a daringfireball link or a mention of the 'Help' link but you have included both as well as a given the history and explained the intuitive basis.

I hope you'll pardon me for playing along a little there. It was a novel experience to be the one receiving the quoting instructions rather than the one typing them out. I liked the feeling of anonymity it gave me and wanted to see if that anonymity could be extended as far as acting out the role of a newcomer seeking further instructions.

Pleased to meet you loup-vaillant and thank you for making my counterfactual newcomer self feel welcome!

Comment author: loup-vaillant 23 April 2012 03:26:56PM 1 point [-]

You got me. Overall, I preffer to judge posts by their content, so I'm glad to learn of your trick.

For the record, even I expected to stop at the Daring Fireball link. I also wrote a bunch of examples, but only then noticed/remembered the "Show help" button. I also erased a sentence about how to show markdown code in markdown (it's rarely useful here, there was the Daring Fireball link, and my real reason for writing it was to show off).

I tend to heavily edit my writings. My most useful heuristic so far is "shorter is better". This very comment benefited from it (let's stop the recursion right there).

Comment author: wedrifid 24 April 2012 12:41:10AM 0 points [-]

You got me. Overall, I preffer to judge posts by their content, so I'm glad to learn of your trick.

It seemed gentler than responding with a direct challenge to the inference behind the presumption.

Comment author: John_Maxwell_IV 23 April 2012 06:02:11AM *  1 point [-]

This may provide more data on what "the AGI community" thinks:

http://wiki.lesswrong.com/wiki/Interview_series_on_risks_from_AI

Comment author: Mitchell_Porter 24 April 2012 03:41:52PM 9 points [-]

It amuses me to think of Eliezer and Pei as like yang and yin. Eliezer has a very yang notion of AI: sharp goals, optimizing, conquering the world in a hurry. Pei's AI doesn't just have limitations - bounded rationality - its very nature is about working with those limitations. And yet, just like the symbol, yin contains yang and yang contains yin: Pei is the one who is forging ahead with a practical AI project, whereas Eliezer is a moral philosopher looking for a mathematical ideal.

Comment author: timtyler 23 April 2012 11:21:35AM *  2 points [-]

Whether we can build a “safe AGI” by giving it a carefully designed “goal system” My answer is negative. It is my belief that an AGI will necessarily be adaptive, which implies that the goals it actively pursues constantly change as a function of its experience, and are not fully restricted by its initial (given) goals.

I don't see Pei distinguishing between instrumental and ultimate goals anywhere. Whereas Luke does do this. Maybe a failure to make that distinction explains the resulting muddle.

Update - 2012-04-23 - it looks as though he does do something similar here:

In terms of source, a goal can be either original or derivative, where the former is imposed on the system, and the latter is produced from the former by the system itself. In the current design of NARS, all input tasks are original goals, and all the derived tasks are derivative goals. In human or animals, all the (biological and evolutionary) innate drives are original goals, and all the other motivations, intentions, and objectives are derivative goals.

...though original / derived doesn't seem to be quite the same idea.

Comment author: timtyler 23 April 2012 01:24:40AM *  5 points [-]

If intelligence turns out to be adaptive (as believed by me and many others), then a “friendly AI” will be mainly the result of proper education, not proper design. There will be no way to design a “safe AI”, just like there is no way to require parents to only give birth to “safe baby” who will never become a criminal.

I don't think that follows. What consumer robot makers will want will be the equivalent of a "safe baby" - who will practically never become a criminal. That will require a tamper-proof brain, and many other safety features. Robot builders won't want to see their robots implicated in crimes. There's no law that says this is impossible, and that's because it is possible.

Machines don't really distinguish between the results of education and apriori knowledge. That's because you can clone adult minds - which effectively blurs the distinction.

Comment author: shminux 23 April 2012 03:49:43AM 1 point [-]

That's because you can clone adult minds - which effectively blurs the distinction.

Clone? Maybe. Hopefully. Create from scratch? Not so sure.

Comment author: timtyler 23 April 2012 10:42:20AM *  2 points [-]

Clone? Maybe. Hopefully. Create from scratch? Not so sure.

I meant that you can clone adult machine minds there.

Comment author: fubarobfusco 23 April 2012 06:23:15AM *  4 points [-]

Luke remarked:

the only people allowed to do philosophy should be those with with primary training in cognitive science, computer science, or mathematics.

I expect that this is meant as a metaphorical remark about the low value of some philosophy, rather than literally as a call for banning anyone from doing philosophy. However, this sort of comment squicks me, especially in this context.

Comment author: lukeprog 23 April 2012 06:52:10AM 5 points [-]

I guess I should have made the joke more explicit. Note that I do not have primary training in cognitive science, computer science, or mathematics, and yet here I am, doing philosophy. :)

Comment author: Normal_Anomaly 23 April 2012 01:26:11PM 7 points [-]

I didn't get the joke either.

Comment author: Emile 23 April 2012 03:10:58PM 6 points [-]

Oh, I read it as an attempt to flatter Pei by taking a cheap shot at Philosophers.

Comment author: ChrisHallquist 27 April 2012 01:57:39AM 0 points [-]

I actually read it and thought, "yeah, that sounds about right." Though re-reading it it's obviously at least hyperbole, though I wouldn't say it's obviously a joke.

Comment author: ChrisHallquist 27 April 2012 02:25:42AM *  3 points [-]

Luke, what do you mean here when you say, "Friendly AI may be incoherent and impossible"?

The Singularity Institute's page "What is Friendly AI?" defines "Friendly AI" as "A "Friendly AI" is an AI that takes actions that are, on the whole, beneficial to humans and humanity." Surely you don't mean to say, "The idea of an AI that takes actions that are, on the whole, beneficial to humans and humanity may be incoherent or impossible"?

Eliezer's paper "Artificial Intelligence as a Positive and Negative Factor in Global Risk" talks about "an AI created with specified motivations." But it's pretty clear that that's not the only thing you and he have in mind, because part of the problem is making sure the motivations we give an AI are the ones we really want to give it.

If you meant neither of those things, what did you mean? "Provably friendly"? "One whose motivations express an ideal extrapolation of our values"? (It seems a flawed extrapolation could still give results that are on the whole beneficial, so this is different than the first definition suggested above.) Or something else?

Comment author: lincolnquirk 23 April 2012 07:18:44AM 3 points [-]

This is great. Pei really impresses me, especially with the linked paper, “The Assumptions of Knowledge and Resources in Models of Rationality”. If you haven't read it, please read it. It clarifies everything Pei is saying and allows you to understand his perspective much better.

That said, I think Luke's final rebuttal was spot on, and I would like to find out whether Pei has changed his mind after reading "Superintelligent Will".

Comment author: timtyler 23 April 2012 07:45:28PM 1 point [-]

It comes across as 20 pages of sour grapes about AIXI :-(

Comment author: Grognor 23 April 2012 01:26:13PM *  4 points [-]

I'm finding these dialogues worthwhile for (so far) lowering my respect for "mainstream" AI researchers.

Pei Wang's definition of intelligence is just "optimization process" in fancy clothes.

His emphasis on raising an AI with prim/proper experience makes me realize how humans can't use our native architecture thinking about AI problems. For so many people, "building a safe AI" just pattern-matches to "raising a child so he becomes a good citizen", even though these tasks have nothing to do with each other. But the analogy is so alluring that there are those who simply can't escape it.

This is a basic mistake. It boggles the mind to see someone who claims to be a mainstream AGI person making it.

Comment author: fubarobfusco 23 April 2012 05:19:00PM 3 points [-]

Pei Wang's definition of intelligence is just "optimization process" in fancy clothes.

I've heard expressions such as "sufficiently powerful optimization process" around LW pretty often, too, especially in the context of sidelining metaphysical questions such as "will AI be 'conscious'?"

Comment author: TheOtherDave 23 April 2012 05:58:22PM 2 points [-]

(nods) I try to use "superhuman optimizer" to refer to superhuman optimizers, both to sidestep irrelevant questions about consciousness and sentience, and to sidestep irrelevant questions about intelligence. It's not always socially feasible, though. (Or at least, I can't always fease it socially.)

Comment author: XiXiDu 23 April 2012 03:02:56PM 3 points [-]

I'm finding these dialogues worthwhile for (so far) lowering my respect for "mainstream" AI researchers.

Think about how ridiculous your comment must sound to them. Some of those people have been researching AI for decades, wrote hundreds of papers that have been cited many thousands of times.

That you just assume that they must be stupid because they disagree with you seems incredible arrogant. They have probably thought about everything you know long before you and dismissed it.

Comment author: wedrifid 23 April 2012 09:03:44PM 5 points [-]

Think about how ridiculous your comment must sound to them.

People expressing disagreement with someone who is confident that they are right and is secure in their own status is going to be perceived by said high status person as foolish (or as an enemy to be crushed). This doesn't mean you should never do so, merely that you will lose the goodwill of the person being criticized if you choose to do so.

Some of those people have been researching AI for decades, wrote hundreds of papers that have been cited many thousands of times.

Note that Grognor gave here a direction of his update based on this conversation. Even if he takes the status of Wang as overwhelmingly strong evidence of the correctness of his position it doesn't mean that the direction of the update based on this particular piece of additional information should not be 'down'. In fact, the more respect Grognor had for the speaker's position prior to hearing him speak, the easier it is for the words to require a downward update. If there wasn't already respect in place the new information wouldn't be surprising.

That you just assume that they must be stupid because they disagree with you seems incredible arrogant.

He didn't do that. Or, at least, Grognor's comment doesn't indicate that he did that. He saw a problem of basic logic in the arguments presented and took that as evidence against the conclusion. If Grognor could not do that it would be essentially pointless for him to evaluate the arguments at all.

Comment author: Grognor 23 April 2012 03:49:33PM *  7 points [-]

Think about how ridiculous your comment must sound to them.

I have no reason to suspect that other people's use of the absurdity heuristic should cause me to reevaluate every argument I've ever seen.

That a de novo AGI will be nothing like a human child in terms of how to make it safe is an antiprediction in that it would take a tremendous amount of evidence to suggest otherwise, and yet Wang just assumes this without having any evidence at all. I can only conclude that the surface analogy is the entire content of the claim.

That you just assume that they must be stupid

If he were just stupid, I'd have no right to be indignant at his basic mistake. He is clearly an intelligent person.

They have probably thought about everything you know long before you and dismissed it.

You are not making any sense. Think about how ridiculous your comment must sound to me.

(I'm starting to hate that you've become a fixture here.)

Comment author: jacob_cannell 16 June 2012 07:21:29PM *  2 points [-]

That a de novo AGI will be nothing like a human child in terms of how to make it safe is an antiprediction in that it would take a tremendous amount of evidence to suggest otherwise, and yet Wang just assumes this without having any evidence at all.

I think this single statement summarizes the huge rift between the narrow specific LW/EY view of AGI and other more mainstream views.

For researchers who are trying to emulate or simulate brain algorithms directly, its self-evidently obvious that the resulting AGI will start like a human child. If they succeed first your 'antiprediction' is trivially false. And then we have researchers like Wang or Goertzel who are pursuing AGI approaches that are not brain-like at all and yet still believe the AGI will learn like a human child and specifically use that analogy.

You can label anything an "antiprediction" and thus convince yourself that you need arbitrary positive evidence to disprove your counterfactual, but in doing so you are really just rationalizing your priors/existing beliefs.

Comment author: XiXiDu 24 April 2012 08:40:27AM *  -2 points [-]

(I'm starting to hate that you've become a fixture here.)

People like you are the biggest problem. I had 4 AI researchers emailing me, after I asked them about AI risks, that they regret having engaged with this community and that they will from now on ignore it because of the belittling attitude of its members.

So congratulations for increasing AI risks by giving the whole community a bad name, idiot.

Comment author: paper-machine 24 April 2012 08:48:50AM 3 points [-]

So congratulations for increasing AI risks by giving the whole community a bad name, idiot.

This is an absolutely unacceptable response on at least three levels.

Comment author: ArisKatsaris 24 April 2012 10:07:52AM *  1 point [-]

And you're complaining about people's belittling attitude, while you call them "idiots" and say stuff like "They have probably thought about everything you know long before you and dismissed it"?

Are you sure those AI researchers weren't referring to you when they were talking about members' belittling attitude?

Comment author: Aharon 24 April 2012 06:56:34PM 6 points [-]

The template XiXiDu used to contact the AI researchers seemed respectful and not at all belittling. I haven't read the interviews themselves yet, but just looking at the one with Brandon Rohrer, his comment

This is an entertaining survey. I appreciate the specificity with which you've worded some of the questions. I don't have a defensible or scientific answer to any of the questions, but I've included some answers below that are wild-ass guesses. You got some good and thoughtful responses. I've been enjoying reading them. Thanks for compiling them.

doesn't sound like he feels belittled by XiXiDu.

Also, I perceived your question as tension-starter, because whatever XiXiDu's faults or virtues, he does seem to respect the opinion of non-LW-researchers more than the average LW-member. I'm not here that often, but I assume that if I noticed that, somebody with a substantially higher Karma would have also noticed that. That makes me think there is a chance that your question wasn't meant as a serious inquiry, but as an attack - which XiXiDu answered to in kind.

Comment author: JGWeissman 24 April 2012 07:37:12PM 4 points [-]

The template XiXiDu used to contact the AI researchers seemed respectful and not at all belittling.

Aris is accusing XiXiDu of being belittling not to the AI researchers, but to the people who disagree with the AI researchers.

Comment author: Aharon 24 April 2012 08:06:59PM *  2 points [-]

Ah, I see. Thanks for the clarification.

Comment author: XiXiDu 24 April 2012 11:39:04AM *  0 points [-]

And you're complaining about people's belittling attitude, while you call them "idiots" and say stuff like "They have probably thought about everything you know long before you and dismissed it"?

Sure. The first comment was a mirror image of his style to show him how it is like when others act the way he does. The second comment was a direct counterstrike against him attacking me and along the lines of what your scriptures teach.

Comment author: ArisKatsaris 24 April 2012 11:44:47AM *  4 points [-]

And what justification do you have for now insulting me by calling the Sequences (I presume) my "scriptures"?

Or are you going to claim that this bit was not meant for an insult and an accusation? I think it very clearly was.

Comment author: JoshuaZ 24 April 2012 04:55:42AM 1 point [-]

If he were just stupid, I'd have no right to be indignant at his basic mistake.

As a general heuristic, if you agree that someone is both intelligent and highly educated, and has made a conclusion that you consider to be a basic mistake, there are a variety of reasonable responses, one of the most obvious of which is to question if the issue in question falls under the category of a basic mistake or even as a mistake at all. Maybe you should update your models?

Comment author: John_Maxwell_IV 24 April 2012 04:46:43AM -1 points [-]

If he were just stupid, I'd have no right to be indignant at his basic mistake.

This is off-topic, but this sentence means nothing to me as a person with a consequentialist morality.

Comment author: Grognor 24 April 2012 01:27:15PM *  2 points [-]

The consequentialist argument is as follows:

Lowering the status of people who make basic mistakes causes them to be less likely to make those mistakes. However, you can't demand that non-intelligent people don't make basic mistakes, as they are going to make them anyway. So demand that smart people do better and maybe they will.

The reasoning is the same as Sark Julian's here/here.

I guess the word "right" threw you off. I am a consequentialist.

Comment author: John_Maxwell_IV 24 April 2012 06:06:48PM 0 points [-]

I'd guess that such status lowering mainly benefits in communicating desired social norms to bystanders. I'm not sure we can expect those whose status is lowered to accept the social norm, or at least not right away.

In general, I'm very uncertain about the best way to persuade people that they could stand to shape up.

Comment author: ciphergoth 28 April 2012 03:31:45PM 0 points [-]

Hadn't seen the antiprediction angle - obvious now you point it out.

I actually applauded this comment. Thank you.

Comment author: Jack 24 April 2012 08:28:58AM 3 points [-]

Some of those people have been researching AI for decades, wrote hundreds of papers that have been cited many thousands of times.

If people are updating correctly they should have already factored in to their beliefs the fact that the SIAI position isn't anywhere near the consensus in university artificial intelligence research. Given the assumption that these people are highly intelligent and well versed in their fields one should probably disbelieve the SIAI position at the outset because one expects that upon hearing from a "mainstream" AI researcher one would learn the reasons why SIAI is wrong. But if you then read a conversation between a "mainstream" AI researcher and an SIAI researcher and the former can't explain why the latter is wrong then you better start updating.

They have probably thought about everything you know long before you and dismissed it.

I'm sure this is true when it comes to, say, programming a particular sort of artificial intelligence. But you are vastly overestimating how much thought scientists and engineers put into broad, philosophical concerns involving their fields. With few exceptions mathematicians don't spend time thinking about the reality of infinite sets, physicists don't spend time thinking about interpretations of quantum mechanics, computer programmers don't spend time thinking about the Church-Turing-Deutsch principle etc.

Comment author: XiXiDu 24 April 2012 08:53:36AM *  2 points [-]

But if you then read a conversation between a "mainstream" AI researcher and an SIAI researcher and the former can't explain why the latter is wrong then you better start updating.

His arguments were not worse than Luke's arguments if you ignore all the links, which he has no reason to read. He said that he does not believe that it is possible to restrict an AI the way that SI does imagine and still produce a general intelligence. He believes that the most promising route is AI that can learn by being taught.

In combination with his doubts about uncontrollable superintelligence, that position is not incoherent. You can also not claim, given this short dialogue, that he did not explain why SI is wrong.

But you are vastly overestimating how much thought scientists and engineers put into broad, philosophical concerns involving their fields.

That's not what I was referring to. I doubt they have thought a lot about AI risks. What I meant is that they have likely thought about the possibility of recursive self-improvement and uncontrollable superhuman intelligence.

If an AI researcher tells you that he believes that AI risks are not a serious issue because they do not believe that AI can get out of control for technical reasons and you reply that they have not thought about AI drives and the philosophical reasons for why superhuman AI will pose a risk, then you created a straw man. Which is the usual tactic employed here.

Comment author: [deleted] 23 April 2012 04:27:20PM 1 point [-]

They have probably thought about everything you know long before you and dismissed it.

And yet they demonstrate that they have not, by not engaging the core arguments made by SI/FHI when they talk about it.

Comment author: jacob_cannell 16 June 2012 07:07:47PM *  0 points [-]

For so many people, "building a safe AI" just pattern-matches to "raising a child so he becomes a good citizen", even though these tasks have nothing to do with each other.

Suppose you build an AI that exactly replicates the developmental algorithms of an infant brain, and you embody it in a perfect virtual body. For this particular type of AI design, the analogy is perfectly exact, and the AI is in fact a child exactly equivalent to a human child.

A specific human brain is a single point in mindspace, but the set of similar architectures extends out into a wider region which probably overlaps highly with much of the useful, viable, accessible space of AGI designs. So the analogy has fairly wide reach.

As an analogy, its hard to see how comparing a young AI to a child is intrinsically worse than comparing the invention of AI to the invention of flight, for example.

Comment author: Peterdjones 21 January 2013 11:06:06AM 0 points [-]

For this particular type of AI design, the analogy is perfectly exact, and the AI is in fact a child exactly equivalent to a human child.

Up until the point that Pinocchio realises he isn't real boy.

Comment author: XiXiDu 23 April 2012 08:31:07AM 3 points [-]

See also Q&A with experts on risks from AI #3, including Pei Wang.

Comment author: semianonymous 23 April 2012 07:42:38PM *  2 points [-]

If you clear away all the noise arising from the fact that this interaction constitutes a clash of tribal factions...

Pei seems to conflate the possibility...

I'm finding these dialogues worthwhile for (so far) lowering my respect for "mainstream" AI researchers...

and so on.

I think it'd be great if SIAI would not lath on the most favourable and least informative interpretation of any disagreement, in precisely the way how e.g. any community around free energy devices does. It'd be also great if Luke allowed for the possibility that Wang (and most other people whom are more intelligent, better educated, and more experienced than Luke) are actually correct, and Luke is completely wrong (or not even wrong).

Comment author: wedrifid 23 April 2012 08:51:20PM 2 points [-]

I think it'd be great if SIAI would not lath on the most favourable and least informative interpretation of any disagreement

The SIAI hasn't seemed to lath on any interpretations. The quote you make and all the interpretations you disagree with here have been done by commenters from the internet that aren't SIAI affiliated. The main thing Luke does that is a disservice to Wang is to post this conversation publicly, thereby embarrassing the guy and lowering his reputation among anyone who finds the reasoning expressed to be poor. But the conversation was supposed to be public and done with Wang's foreknowledge - presumably the arguments he used he actually wants to be associated with.

It'd be also great if Luke allowed for the possibility that Wang (and most other people whom are more intelligent, better educated, and more experienced than Luke) are actually correct, and Luke is completely wrong (or not even wrong).

As Grognor said (in the quote you made) this particular conversation served to significantly lower the perceived likelyhood that those things are correct. And this could become a real problem if it happens too often. Being exposed to too many bad arguments for a position can serve to bias the reader in the opposite direction to what has been argued. We need to find representatives from "mainstream AI researchers" that don't make the kind of simple errors of reasoning that we see here. Presumably they exist?

Comment author: JoshuaZ 24 April 2012 01:52:05AM 1 point [-]

more intelligent, better educated, and more experienced

2 of these 3 seems to be clearly the case. I'm unsure how you are getting more intelligent. Your point may be valid completely without the intelligence bit in that intelligent people can easily be deeply mistaken about areas they don't have much education, and one sees that not that infrequently. I'm am however curious how you are making the intelligence determination in question.

Comment author: semianonymous 24 April 2012 04:01:11AM -2 points [-]

I'm unsure how you are getting more intelligent.

I'm unsure how you are not. Every single proxy for intelligence indicates a fairly dramatic gap in intelligence in favour of Wang. Of course for politeness sake we assume that they would be at least equally intelligent, and for phyg sake that Luke would be more intelligent, but it is simply very, very, very unlikely.

Comment author: JoshuaZ 24 April 2012 04:06:47AM 0 points [-]

Every single proxy for intelligence indicates a fairly dramatic gap in intelligence in favour of Wang

Can you state explicitly what proxies you are using here that you think indicate a dramatic gap?

Comment author: semianonymous 24 April 2012 04:17:31AM *  3 points [-]

Accomplishments of all kinds, the position, the likelihood that Wang has actually managed to move from effectively lower class (third world) to upper class (but I didn't look up where he's from, yet), etc.

What proxies do you think would indicate Luke is more intelligent? I can't seem to think of any.

Comment author: JoshuaZ 24 April 2012 04:48:13AM *  6 points [-]

Accomplishments of all kinds, the position, the likelihood that Wang has actually managed to move from effectively lower class (third world) to upper class (but I didn't look up where he's from, yet), etc.

Wang is accomplished to the point where one can immediately see it simply from glancing at his CV. However, accomplishments are only a rough measure of intelligence. By some metrics, Conscientiousness is a better predictor of success than raw intelligence, and by many metrics it is at least as good a predictor. Relying on academic success as a metric of intelligence isn't that reliable unless one is doing something like comparing the very top in a field. This also makes little sense given that Luke isn't a member of academia.

The claim about the third world is puzzling- Wang is Chinese (a fact that I would think would be obvious from his name, and took me two seconds to verify by looking at his CV) and China has never been considered third world, but rather was (when the term made more sense) second world. Moreover, this isn't just an argument over the meaning of words- China's GDP per a capita, average education level, average literacy level[1], or almost any other metric you choose is far higher than that of most countries classically considered to be in the third world.

Wang is also older than Luke. Wang finished his undergraduate degree in 1983, so he's approximately in his early fifties now. Pei Wang has therefore had far more time to accomplish things. So simply lining up their accomplishment levels doesn't work. (Although Wang clearly does have some accomplishments at a fairly young age, such as his thesis being selected for an Outstanding Dissertation Award by his university.)

What proxies do you think would indicate Luke is more intelligent? I can't seem to think of any.

I'm not sure why this question is being asked. I'm not aware of any either but it really doesn't have much to do with the matter at hand. You've claimed not just that Wang is likely to be more intelligent but that "Every single proxy for intelligence indicates a fairly dramatic gap in intelligence"- that requires a lot more than simply not having any obvious pointers for Luke to be smarter. Overall, I'm deeply unconvinced that either one is more intelligent. This isn't an issue of Luke being more intelligent. This is an issue of very little data in general.

[1] Some of the entries in that list are measured with different metrics, so this isn't a perfect comparison.

Comment author: semianonymous 24 April 2012 04:59:59AM *  -3 points [-]

There is very little data on Luke and that is a proxy for Luke being less intelligent, dramatically so. It is instrumental to Luke's goals to provide such data. On the second world or third world that is irrelevant semantics.

edit: and as rather strong evidence that Luke is approximately as intelligent as the least intelligent version of Luke that can look the same to us, it suffices to cite normal distribution of intelligence.

Comment author: JoshuaZ 24 April 2012 05:04:44AM *  2 points [-]

That reply essentially ignores almost every comment I made. I'm particularly curious whether you are in agreement that Pei Wang isn't from a third world country? Does that cause any update at all for your estimates?

There is very little data on Luke and that is a proxy for Luke being less intelligent, dramatically so.

Also, if we go back in time 20 years, so that Pei Wang would be about the same age Luke is now, do you think you'd have an accomplishment list for Pei Wang that was substantially longer than Luke's current one? If so, how does that impact your claim?

Comment author: semianonymous 24 April 2012 05:26:20AM *  4 points [-]

I apologise for my unawareness that you call China second world. It is still the case that it is very difficult to move from China to US.

Also, if we go back in time 20 years, so that Pei Wang would be about the same age Luke is now, do you think you'd have an accomplishment list for Pei Wang that was substantially longer than Luke's current one? If so, how does that impact your claim?

If we move back 20 years, it is 1992, and Pei Wang has already been a lecturer in China then moved to Indiana University. Length of the accomplishment list is a poor proxy, difficulty is important. As I explained in the edit, you shouldn't forget about Bell's curve. No evidence for intelligence is good evidence of absence, on the IQ>100 side of normal distribution.

Comment author: JoshuaZ 24 April 2012 05:40:10AM 0 points [-]

As I explained in the edit, you shouldn't forget about Bell's curve. No evidence for intelligence is good evidence of absence, on the IQ>100 side of normal distribution.

Ah, that's what you meant by the other remark. In that case, this isn't backing up claimed prior proxies and is a new argument. Let's be clear on that. So how valid is this? I don't think this is a good argument at all. Anyone who has read what Luke has to say or interacted with Luke can tell pretty strongly that Luke is on the right side of the Bell curve. Sure, if I pick a random person the chance that they are as smart as Pei Wang is tiny, but that's not the case here.

It is still the case that it is very difficult to move from China to US

There are a lot of Chinese academics who come to the United States. So what do you mean by very difficult?

If we move back 20 years, it is 1992, and Pei Wang has already been a lecturer in China then moved to Indiana University.

He doesn't have his PhD at that point. He gets that at Indiana. I can't tell precisely from his CV what he means by lecturer, but at least in the US it often means a position primarily given for teaching rather than research. Given that he didn't have a doctorate at the time, it is very likely that it means something similar, what we might call an adjunct here. That it isn't a very good demonstration of intelligence at all. Luke has in his time run a popular blog that has been praised for its clarity and good writing. And you still haven't addressed the issue that Luke was never trying to go into academia.

Comment author: JoshuaZ 24 April 2012 05:20:52AM *  0 points [-]

Replying in a second comment to the parts you edited in (to keep conversation flow clear and also so you see this remark):

On the second world or third world that is irrelevant semantics.

I outlined precisely why this wasn't just a semantic issue. China and most Chinese citizens are pretty well off and they have access to a decent education system. This isn't a semantic issue. A moderately smart random Chinese citizen has pretty decent chances at success.

and as rather strong evidence that Luke is approximately as intelligent as the least intelligent version of Luke that can look the same to us, it suffices to cite normal distribution of intelligence.

I don't understand this comment. I can't parse it in any way that makes sense. Can you expand/clarify on this remark? Also, is this to be understood as a new argument for a dramatic intelligence gap and not an attempt to address the previously listed proxies?

Comment author: wedrifid 24 April 2012 06:14:04AM -2 points [-]

Wang is also older than Luke. Wang finished his undergraduate degree in 1983, so he's approximately in his early fifties now. Pei Wang has therefore had far more time to accomplish things.

And, it must be noted, more time to crystallize intuitions formed based off the common sense from yesteryear.

Comment author: JoshuaZ 24 April 2012 06:25:37AM 0 points [-]

That isn't relevant for the immediate issue of intelligence evaluation. It may be relevant to the general point at hand, but it sounds worryingly like a fully general counterargument.

Comment author: wedrifid 24 April 2012 06:37:42AM *  -1 points [-]

That isn't relevant for the immediate issue of intelligence evaluation.

It was a tangent of general interest to the progress of science. It could have been made purely as a relevant-to-intelligence-evaluation point if it were expanded by pointing to the well understood relationship of fluid and crystallized intelligence as they change over time.

It may be relevant to the general point at hand, but it sounds worryingly like a fully general counterargument.

It is merely something that tempers the degree to which the fully general argument "This person is more experienced and has collected more prestige therefore he is right" should be given weight. It would become a 'fully general counterargument' when people started using "nah, she's old" in a general context. When used specifically when evaluating the strength of the evidence indicated by prestige it is simply one of the relevant factors under consideration.

There is a world of difference between a minor point of general relevance to the evaluation of a specific kind of evidence and a "fully general counter-argument". The abuse of the former would be required for the latter charge to be justified - and that isn't the case here.

Comment author: JoshuaZ 24 April 2012 06:38:35AM *  0 points [-]

Good point.

Comment author: ArisKatsaris 24 April 2012 10:56:10AM 4 points [-]

If accomplishments is the only proxy you use to evaluate their relative intelligence, then it would have been all-around better if you had said "more accomplished" rather than "more intelligent", as it's more precise, less controversial, and doesn't confuse fact with inference.

Comment author: semianonymous 24 April 2012 06:45:45PM *  -2 points [-]

It also does not present valid inference. Ideally, you're right but in practice people do not make the inferences they do not like.

Comment author: ArisKatsaris 24 April 2012 06:52:07PM *  2 points [-]

It also does not present valid inference.

If you wanted to present the inference, then present it as an inference.

e.g. "more accomplished (and thus I conclude more intelligent)" would have been vastly better than what you did, which was to just present your conclusion in a manner that would inevitably bait others to dispute it/take offense against it.

Comment author: cousin_it 23 April 2012 01:15:11AM *  2 points [-]

Eliezer's epiphany about precision, which I completely subscribe to, negates most of Pei's arguments for me.

Comment author: XiXiDu 23 April 2012 10:02:15AM 2 points [-]

Eliezer's epiphany about precision, which I completely subscribe to, negates most of Pei's arguments for me.

I have read the post and I still don't understand what you mean.

Comment author: cousin_it 23 April 2012 06:11:54PM *  2 points [-]

As I understand it, Eliezer's concept of precision involves trying to find formalizations that are provably unique or optimal in some sense, not just formalizations that work. For example, Bayesianism is a unique solution under the conditions of Cox's theorem. One of Pei's papers points out flaws in Bayesianism and proposes an alternative approach, but without any proof of uniqueness. I see that as an avoidable mistake.

Comment author: Wei_Dai 23 April 2012 08:36:05PM *  8 points [-]

I guess Pei's intuition is that a proof of uniqueness or optimality under unrealistic assumptions is of little practical value, and doing such proofs under realistic assumptions is unfeasible compared to the approach he is taking.

ETA: When you write most kinds of software, you don't first prove that your design is optimal or unique, but just start with something that you intuitively think would work, and then refine it by trial and error. Why shouldn't this work for AGI?

ETA 2: In case it wasn't clear, I'm not advocating that we build AGIs by trial and error, but just trying to explain what Pei is probably thinking, and why cousin_it's link isn't likely to be convincing for him.

Comment author: TheOtherDave 23 April 2012 09:08:58PM -1 points [-]

If one isn't concerned about the AGI's ability to either (a) be able out-of-the-box to either successfully subvert the testing mechanisms being applied to it or successfully neutralize whatever mechanisms are in place to deal with it if it "fails" those tests, or (b) self-improve rapidly enough to achieve that state before those testing or dealing-with-failures mechanisms apply, then sure, a sufficiently well-designed test harness around some plausible-but-not-guaranteed algorithms will work fine, as it does for most software.

Of course, if one is concerned about an AGI's ability to do either of those things, one may not wish to rely on such a test harness.

It seems to follow from this that some kind of quantification of what kinds of algorithms can do either of those things, and whether there's any way to reliably determine whether a particular algorithm falls into that set prior to implementing it, might allow AGI developers to do trial-and-error work on algorithms that provably don't meet that standard would be one way of making measurable progress without arousing the fears of those who consider FOOMing algorithms a plausible existential risk.

Of course, that doesn't have the "we get it right and then everything is suddenly better" aspect of successfully building a FOOMing FAI... it's just research and development work, the same sort of incremental collective process that has resulted in, well, pretty much all human progress to date.

Comment author: loup-vaillant 23 April 2012 01:17:48PM *  0 points [-]

I guess he was talking about the kind of precision more specific to AI, which goes like "compared to the superintelligence space, the Friendly AI space is a tiny dot. We should aim precisely, at the first try, or else".

And because the problem is impossible to solve, you have to think precisely in the first place, or you won't be able to aim precisely at the first try. (Because whatever Skynet we build won't give us the shadow of a second chance).

Comment author: timtyler 23 April 2012 06:11:13PM 2 points [-]

I guess he was talking about the kind of precision more specific to AI, which goes like "compared to the superintelligence space, the Friendly AI space is a tiny dot. We should aim precisely, at the first try, or else".

Compared to the space of possible 747 component configurations, the space of systems representing working 747s is tiny. We should aim precisely, at the first try, or else!

Comment author: XiXiDu 23 April 2012 01:35:42PM 1 point [-]

I guess he was talking about the kind of precision more specific to AI, which goes like "compared to the superintelligence space, the Friendly AI space is a tiny dot.

But that stance makes assumptions that he does not share, as he does not believe that AGI will become uncontrollable.

Comment author: Stuart_Armstrong 24 April 2012 04:58:55PM *  2 points [-]

Pei's point that FAI is hard/impossible seems to me to be an argument that we should be even more keen to stop AGI research. He does a good argument that FAI doesn't work, but hasn't got anything to substitute in its place ("educating the AI" might be the only way to go, but that doesn't mean it's a good way to go).

Comment author: shminux 24 April 2012 06:29:02PM 2 points [-]

It seems to me that Pei's point is that it is too early to know whether FAI is possible, and further research is needed to determine this. There may come a day when the only prudent course of action is to stop the AI research cold in the name of survival of the human race, but today is not that day.

Comment author: Stuart_Armstrong 25 April 2012 09:28:59AM 1 point [-]

It seems to me that Pei's point is that it is too early to know whether FAI is possible, and further research is needed to determine this

Agreed. But he never engages with the idea of pointing the field in a different direction, of prioritising certain types of research. He concludes that the field is fine exactly as it is now, and that the researchers should all be left alone.

I think we can detect a strong hint of status quo bias here. Not enough to dismiss his points, but enough to question them. If he'd concluded "we need less safety research than now" or something, I'd have respected his opinion more.

Comment author: shminux 25 April 2012 03:09:34PM 2 points [-]

But he never engages with the idea of pointing the field in a different direction, of prioritising certain types of research.

Why would he? According to his views, the field is still in its infancy, stifling its natural development in any way would be a bad idea.

Comment author: Luke_A_Somers 23 April 2012 03:59:50PM *  2 points [-]

Pei seems to conflate the possibility of erroneous beliefs with the possibility of unfortunate (for us) goals. The Assumption of Insufficient Knowledge and Resources isn't what FAI is about, yet you get statements like

As I mentioned above, the goal system of an adaptive system evolves as a function of the system’s experience. No matter what initial goals are implanted, under AIKR the derived goals are not necessarily their logical implications, which is not necessarily a bad thing (the humanity is not a logical implication of the human biological nature, neither), though it means the designer has no full control to it (unless the designer also fully controls the experience of the system, which is practically impossible). See “The self-organization of goals” for detailed discussion.

Okay, so no one, not even superintelligent AI, is infallible. An AI may take on misguided instrumental goals. Yup. No way around that. That's totally absolutely missing the point.

Unless of course you think that a non-negligible portion of uFAI outcomes are where it does something horrible to us by accident while only wanting the best for us and having a clear, accurate conception of what that is.

Comment author: fubarobfusco 23 April 2012 06:27:56AM 0 points [-]

Pei remarked:

In scientific theories, broader notions are not always better. In this context, a broad notion may cover too many diverse approaches to provide any non-trivial conclusion.

Sounds like Eliezer's advice to be specific, doesn't it? Or even the virtue of narrowness.

Comment author: [deleted] 23 April 2012 04:32:45PM 4 points [-]

Yes it sounds like that, the same way as "Scientific theories should be narrow, mr. Newton. you should focus on just falling objects instead of trying to broaden your theory to the entire cosmos."

Comment author: Jayson_Virissimo 23 April 2012 07:44:36AM 0 points [-]

Not really, no.

Comment author: p4wnc6 29 April 2012 10:21:23PM *  1 point [-]

I think the most glaring problem I could detect with Pei's position is captured in this quotation:

Therefore, to control the morality of an AI mainly means to educate it properly (i.e., to control its experience, especially in its early years). Of course, the initial goals matters, but it is wrong to assume that the initial goals will always be the dominating goals in decision making processes.

This totally dodge's Luke's point that we don't have a clue what such moral education would be like because we don't understand these things about people. For this specific point of Pei's to be taken seriously, we'd have to believe that (1) AGI's will be built, (2) They will be built in such a way as to accept their sensory inputs in modes that are exceedingly similar to human sensory perception (which we do not understand very well at all), (3) The time scale of the very-human-perceptive AGI's cognition will also be extremely similar to human cognition.

Then we could "teach" the AGI how to be moral much like we teach a child our favorite cultural and ethical norms. But I don't see any reason why (1), (2), or (3) should be likely, let alone why their intersection should be likely.

I think Pei is suffering an unfortunate mind projection fallacy here. He seems to have in his mind humanoid robot with AGI software for brains, that has similar sensory modalities, updates its brain state at a similar rate as a human, steers its attention mechanisms in similar ways. This is outrageously unlikely for something that didn't evolve on the savanna, that isn't worried about whether predators are coming around the corner, and isn't really hungry and looking for a store of salt/fat/sugar. This would only be likely if something like connectomics and emulations becomes evident as the most likely path to digital AGI. But it seems like Pei's default assumption.

To boot, suppose we did build AGI that "think slowly" like people do, and could be "taught" morals in a manner similar to people. Why wouldn't defense organizations and governments grab those first copies and modify them to think faster? There would probably be a tech war, or a race at least, and as the mode of cognition was optimized out of the human-similar regime, control over moral developments would be lost very quickly.

Lastly, if an AGI thinks much faster than a human (I believe this is likely to happen very soon after AGI is created), then even if it's long-term goal structure is free to change in response to the environment, it won't matter on time scales that humans care about. If it just wants to play Super Mario Brothers, and it thinks 10,000+ times faster than we do, we won't have an opportunity to convince it to listen to our moral teachings. By the time we say "Treat others as..." we'd be dead. Specifically, Pei says,

it is wrong to assume that the initial goals will always be the dominating goals in decision making processes.

But in terms of time scales that matter for human survival, initial goals will certainly dominate. It's pure mind projection fallacy to imagine an AGI that has slow-moving adjustments to its goals, in such a way that we can easily control and guide them. Belief that the initial state doesn't much matter is a really dangerous assumption.

That someone as intelligent and well-educated on this topic as Pei can believe that assumption without even acknowledging that it's an assumption, let alone a most likely unjustified one, is very terrifying to me.

Comment author: Peterdjones 21 January 2013 12:23:00PM *  1 point [-]

This totally dodge's Luke's point that we don't have a clue what such moral education would be like because we don't understand these things about people.

Says who? We can't mass produce saints, but we know that people from stable well-resourced homes tend not to be criminal.

They will be built in such a way as to accept their sensory inputs in modes that are exceedingly similar to human sensory perception (which we do not understand very well at all)

There's a lot of things we don't understand. We don't know how to build AI's with human-style intelligene at switch-on, so Pei's assumption that training will be required is probably on the money.

The time scale of the very-human-perceptive AGI's cognition will also be extremely similar to human cognition.

We can;t make it as a fast as we like, but we can make it as slow as we like, If we need to train an AGI, and if it's clock speed is hindering the process, then it is trivial to reduce it

I think Pei is suffering an unfortunate mind projection fallacy here. He seems to have in his mind humanoid robot with AGI software for brains, that has similar sensory modalities, updates its brain state at a similar rate as a human, steers its attention mechanisms in similar ways. This is outrageously unlikely for something that didn't evolve on the savanna,

Given the training assumption, it is likely: we will only be able to train an AI into humanlike intelligence if it is humanlike ITFP. Unhumanlike AIs will be abortive projects.

That someone as intelligent and well-educated on this topic as Pei can believe that assumption without even acknowledging that it's an assumption, let alone a most likely unjustified one, is very terrifying to me.

I think his assumptions make more sense that the LessWrongian assumption of AIs that are intelligent at boot-up.

Comment author: Peterdjones 21 January 2013 10:52:37AM 1 point [-]

This totally dodge's Luke's point that we don't have a clue what such moral education would be like because we don't understand these things about people.

The ordinary, average person isn't a psychopath, so presumable the ordinary average education is good enough to avoid psychopathy even if it does';t create saints.

Comment author: siodine 25 April 2012 04:12:32AM 1 point [-]

Better than Goetzel but why didn't you put self-improving AI on the table with the rest of the propositions? That's a fundamental piece of the puzzle in understanding Pei's position. It could be that he thinks a well taught AI is safe enough because it won't self-modify.

Comment author: jacoblyles 31 July 2012 06:47:12PM 0 points [-]

If an AGI research group were close to success but did not respect friendly AI principles, should the government shut them down?

Comment author: Mitchell_Porter 18 August 2012 05:24:37AM 1 point [-]

Let's try an easier question first. If someone is about to create Skynet, should you stop them?

Comment author: jacoblyles 18 August 2012 05:32:54AM *  1 point [-]

The principles espoused by the majority on this site can be used to justify some very, very bad actions.

1) The probability of someone inventing AI is high

2) The probability of someone inventing unfriendly AI if they are not associated with SIAI is high

3) The utility of inventing unfriendly AI is negative MAXINT

4) "Shut up and calculate" - trust the math and not your gut if your utility calculations tell you to do something that feels awful.

It's not hard to figure out that Less Wrong's moral code supports some very, unsavory, actions.

Comment author: Mitchell_Porter 18 August 2012 06:07:44AM 1 point [-]

Your original question wasn't about LW. Before we turn this into a debate about finetuning LW's moral code, shall we consider the big picture? It's 90 years since the word "robot" was introduced, in a play which already featured the possibility of a machine uprising. It's over 50 years since "artificial intelligence" was introduced as a new academic discipline. We already live in a world where one state can use a computer virus to disrupt the strategic technical infrastructure of another state. The quest for AI, and the idea of popular resistance to AI, have already been out there in the culture for years.

Furthermore, the LW ethos is pro-AI as well as anti-AI. But if they feel threatened by AIs, the average person will just be anti-AI. The average person isn't pining for the singularity, but they do want to live. Imagine something like the movement to stop climate change, but it's a movement to stop the singularity. Such a movement would undoubtedly appropriate any useful sentiments it found here, but its ethos and organization would be quite different. You should be addressing yourself to this future anti-AI, pro-human movement, and explaining to them why anyone who works on any form of AI should be given any freedom to do so at all.