Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Reply to Holden on 'Tool AI'

92 Post author: Eliezer_Yudkowsky 12 June 2012 06:00PM

I begin by thanking Holden Karnofsky of Givewell for his rare gift of his detailed, engaged, and helpfully-meant critical article Thoughts on the Singularity Institute (SI). In this reply I will engage with only one of the many subjects raised therein, the topic of, as I would term them, non-self-modifying planning Oracles, a.k.a. 'Google Maps AGI' a.k.a. 'tool AI', this being the topic that requires me personally to answer.  I hope that my reply will be accepted as addressing the most important central points, though I did not have time to explore every avenue.  I certainly do not wish to be logically rude, and if I have failed, please remember with compassion that it's not always obvious to one person what another person will think was the central point.

Luke Mueulhauser and Carl Shulman contributed to this article, but the final edit was my own, likewise any flaws.

Summary:

Holden's concern is that "SI appears to neglect the potentially important distinction between 'tool' and 'agent' AI." His archetypal example is Google Maps:

Google Maps is not an agent, taking actions in order to maximize a utility parameter. It is a tool, generating information and then displaying it in a user-friendly manner for me to consider, use and export or discard as I wish.

The reply breaks down into four heavily interrelated points:

First, Holden seems to think (and Jaan Tallinn doesn't apparently object to, in their exchange) that if a non-self-modifying planning Oracle is indeed the best strategy, then all of SIAI's past and intended future work is wasted.  To me it looks like there's a huge amount of overlap in underlying processes in the AI that would have to be built and the insights required to build it, and I would be trying to assemble mostly - though not quite exactly - the same kind of team if I was trying to build a non-self-modifying planning Oracle, with the same initial mix of talents and skills.

Second, a non-self-modifying planning Oracle doesn't sound nearly as safe once you stop saying human-English phrases like "describe the consequences of an action to the user" and start trying to come up with math that says scary dangerous things like (he translated into English) "increase the correspondence between the user's belief about relevant consequences and reality".  Hence why the people on the team would have to solve the same sorts of problems.

Appreciating the force of the third point is a lot easier if one appreciates the difficulties discussed in points 1 and 2, but is actually empirically verifiable independently:  Whether or not a non-self-modifying planning Oracle is the best solution in the end, it's not such an obvious privileged-point-in-solution-space that someone should be alarmed at SIAI not discussing it.  This is empirically verifiable in the sense that 'tool AI' wasn't the obvious solution to e.g. John McCarthy, Marvin Minsky, I. J. Good, Peter Norvig, Vernor Vinge, or for that matter Isaac Asimov.  At one point, Holden says:

One of the things that bothers me most about SI is that there is practically no public content, as far as I can tell, explicitly addressing the idea of a "tool" and giving arguments for why AGI is likely to work only as an "agent."

If I take literally that this is one of the things that bothers Holden most... I think I'd start stacking up some of the literature on the number of different things that just respectable academics have suggested as the obvious solution to what-to-do-about-AI - none of which would be about non-self-modifying smarter-than-human planning Oracles - and beg him to have some compassion on us for what we haven't addressed yet.  It might be the right suggestion, but it's not so obviously right that our failure to prioritize discussing it reflects negligence.

The final point at the end is looking over all the preceding discussion and realizing that, yes, you want to have people specializing in Friendly AI who know this stuff, but as all that preceding discussion is actually the following discussion at this point, I shall reserve it for later.

1.  The math of optimization, and the similar parts of a planning Oracle.

What does it take to build a smarter-than-human intelligence, of whatever sort, and have it go well?

A "Friendly AI programmer" is somebody who specializes in seeing the correspondence of mathematical structures to What Happens in the Real World. It's somebody who looks at Hutter's specification of AIXI and reads the actual equations - actually stares at the Greek symbols and not just the accompanying English text - and sees, "Oh, this AI will try to gain control of its reward channel," as well as numerous subtler issues like, "This AI presumes a Cartesian boundary separating itself from the environment; it may drop an anvil on its own head." Similarly, working on TDT means e.g. looking at a mathematical specification of decision theory, and seeing "Oh, this is vulnerable to blackmail" and coming up with a mathematical counter-specification of an AI that isn't so vulnerable to blackmail.

Holden's post seems to imply that if you're building a non-self-modifying planning Oracle (aka 'tool AI') rather than an acting-in-the-world agent, you don't need a Friendly AI programmer because FAI programmers only work on agents. But this isn't how the engineering skills are split up. Inside the AI, whether an agent AI or a planning Oracle, there would be similar AGI-challenges like "build a predictive model of the world", and similar FAI-conjugates of those challenges like finding the 'user' inside an AI-created model of the universe.  The insides would look a lot more similar than the outsides.  An analogy would be supposing that a machine learning professional who does sales optimization for an orange company couldn't possibly do sales optimization for a banana company, because their skills must be about oranges rather than bananas.

Admittedly, if it turns out to be possible to use a human understanding of cognitive algorithms to build and run a smarter-than-human Oracle without it being self-improving - this seems unlikely, but not impossible - then you wouldn't have to solve problems that arise with self-modification.  But this eliminates only one dimension of the work.  And on an even more meta level, it seems like you would call upon almost identical talents and skills to come up with whatever insights were required - though if it were predictable in advance that we'd abjure self-modification, then, yes, we'd place less emphasis on e.g. finding a team member with past experience in reflective math, and wouldn't waste (additional) time specializing in reflection.  But if you wanted math inside the planning Oracle that operated the way you thought it did, and you wanted somebody who understood what could possibly go wrong and how to avoid it, you would need to make a function call to the same sort of talents and skills to build an agent AI, or an Oracle that was self-modifying, etc.

2.  Yes, planning Oracles have hidden gotchas too.

"Tool AI" may sound simple in English, a short sentence in the language of empathically-modeled agents — it's just "a thingy that shows you plans instead of a thingy that goes and does things." If you want to know whether this hypothetical entity does X, you just check whether the outcome of X sounds like "showing someone a plan" or "going and doing things", and you've got your answer.  It starts sounding much scarier once you try to say something more formal and internally-causal like "Model the user and the universe, predict the degree of correspondence between the user's model and the universe, and select from among possible explanation-actions on this basis."

Holden, in his dialogue with Jaan Tallinn, writes out this attempt at formalizing:

Here's how I picture the Google Maps AGI ...

utility_function = construct_utility_function(process_user_input());

foreach $action in $all_possible_actions {

$action_outcome = prediction_function($action,$data);

$utility = utility_function($action_outcome);

if ($utility > $leading_utility) { $leading_utility = $utility;

$leading_action = $action; }

}

report($leading_action);

construct_utility_function(process_user_input()) is just a human-quality function for understanding what the speaker wants. prediction_function is an implementation of a human-quality data->prediction function in superior hardware. $data is fixed (it's a dataset larger than any human can process); same with $all_possible_actions. report($leading_action) calls a Google Maps-like interface for understanding the consequences of $leading_action; it basically breaks the action into component parts and displays predictions for different times and conditional on different parameters.

Google Maps doesn't check all possible routes. If I wanted to design Google Maps, I would start out by throwing out a standard planning technique on a connected graph where each edge has a cost function and there's a good heuristic measure of the distance, e.g. A* search. If that was too slow, I'd next try some more efficient version like weighted A* (or bidirectional weighted memory-bounded A*, which I expect I could also get off-the-shelf somewhere). Once you introduce weighted A*, you no longer have a guarantee that you're selecting the optimal path.  You have a guarantee to within a known factor of the cost of the optimal path — but the actual path selected wouldn't be quite optimal. The suggestion produced would be an approximation whose exact steps depended on the exact algorithm you used. That's true even if you can predict the exact cost — exact utility — of any particular path you actually look at; and even if you have a heuristic that never overestimates the cost.

The reason we don't have God's Algorithm for solving the Rubik's Cube is that there's no perfect way of measuring the distance between any two Rubik's Cube positions — you can't look at two Rubik's cube positions, and figure out the minimum number of moves required to get from one to another. It took 15 years to prove that there was a position requiring at least 20 moves to solve, and then another 15 years to come up with a computer algorithm that could solve any position in at most 20 moves, but we still can't compute the actual, minimum solution to all Cubes ("God's Algorithm"). This, even though we can exactly calculate the cost and consequence of any actual Rubik's-solution-path we consider.

When it comes to AGI — solving general cross-domain "Figure out how to do X" problems — you're not going to get anywhere near the one, true, optimal answer. You're going to — at best, if everything works right — get a good answer that's a cross-product of the "utility function" and all the other algorithmic properties that determine what sort of answer the AI finds easy to invent (i.e. can be invented using bounded computing time).

As for the notion that this AGI runs on a "human predictive algorithm" that we got off of neuroscience and then implemented using more computing power, without knowing how it works or being able to enhance it further: It took 30 years of multiple computer scientists doing basic math research, and inventing code, and running that code on a computer cluster, for them to come up with a 20-move solution to the Rubik's Cube. If a planning Oracle is going to produce better solutions than humanity has yet managed to the Rubik's Cube, it needs to be capable of doing original computer science research and writing its own code. You can't get a 20-move solution out of a human brain, using the native human planning algorithm. Humanity can do it, but only by exploiting the ability of humans to explicitly comprehend the deep structure of the domain (not just rely on intuition) and then inventing an artifact, a new design, running code which uses a different and superior cognitive algorithm, to solve that Rubik's Cube in 20 moves. We do all that without being self-modifying, but it's still a capability to respect.

And I'm not even going into what it would take for a planning Oracle to out-strategize any human, come up with a plan for persuading someone, solve original scientific problems by looking over experimental data (like Einstein did), design a nanomachine, and so on.

Talking like there's this one simple "predictive algorithm" that we can read out of the brain using neuroscience and overpower to produce better plans... doesn't seem quite congruous with what humanity actually does to produce its predictions and plans.

If we take the concept of the Google Maps AGI at face value, then it actually has four key magical components.  (In this case, "magical" isn't to be taken as prejudicial, it's a term of art that means we haven't said how the component works yet.)  There's a magical comprehension of the user's utility function, a magical world-model that GMAGI uses to comprehend the consequences of actions, a magical planning element that selects a non-optimal path using some method other than exploring all possible actions, and a magical explain-to-the-user function.

report($leading_action) isn't exactly a trivial step either. Deep Blue tells you to move your pawn or you'll lose the game. You ask "Why?" and the answer is a gigantic search tree of billions of possible move-sequences, leafing at positions which are heuristically rated using a static-position evaluation algorithm trained on millions of games. Or the planning Oracle tells you that a certain DNA sequence will produce a protein that cures cancer, you ask "Why?", and then humans aren't even capable of verifying, for themselves, the assertion that the peptide sequence will fold into the protein the planning Oracle says it does.

"So," you say, after the first dozen times you ask the Oracle a question and it returns an answer that you'd have to take on faith, "we'll just specify in the utility function that the plan should be understandable."

Whereupon other things start going wrong. Viliam_Bur, in the comments thread, gave this example, which I've slightly simplified:

Example question: "How should I get rid of my disease most cheaply?" Example answer: "You won't. You will die soon, unavoidably. This report is 99.999% reliable". Predicted human reaction: Decides to kill self and get it over with. Success rate: 100%, the disease is gone. Costs of cure: zero. Mission completed.

Bur is trying to give an example of how things might go wrong if the preference function is over the accuracy of the predictions explained to the human— rather than just the human's 'goodness' of the outcome. And if the preference function was just over the human's 'goodness' of the end result, rather than the accuracy of the human's understanding of the predictions, the AI might tell you something that was predictively false but whose implementation would lead you to what the AI defines as a 'good' outcome. And if we ask how happy the human is, the resulting decision procedure would exert optimization pressure to convince the human to take drugs, and so on.

I'm not saying any particular failure is 100% certain to occur; rather I'm trying to explain - as handicapped by the need to describe the AI in the native human agent-description language, using empathy to simulate a spirit-in-a-box instead of trying to think in mathematical structures like A* search or Bayesian updating - how, even so, one can still see that the issue is a tad more fraught than it sounds on an immediate examination.

If you see the world just in terms of math, it's even worse; you've got some program with inputs from a USB cable connecting to a webcam, output to a computer monitor, and optimization criteria expressed over some combination of the monitor, the humans looking at the monitor, and the rest of the world. It's a whole lot easier to call what's inside a 'planning Oracle' or some other English phrase than to write a program that does the optimization safely without serious unintended consequences. Show me any attempted specification, and I'll point to the vague parts and ask for clarification in more formal and mathematical terms, and as soon as the design is clarified enough to be a hundred light years from implementation instead of a thousand light years, I'll show a neutral judge how that math would go wrong. (Experience shows that if you try to explain to would-be AGI designers how their design goes wrong, in most cases they just say "Oh, but of course that's not what I meant." Marcus Hutter is a rare exception who specified his AGI in such unambiguous mathematical terms that he actually succeeded at realizing, after some discussion with SIAI personnel, that AIXI would kill off its users and seize control of its reward button. But based on past sad experience with many other would-be designers, I say "Explain to a neutral judge how the math kills" and not "Explain to the person who invented that math and likes it.")

Just as the gigantic gap between smart-sounding English instructions and actually smart algorithms is the main source of difficulty in AI, there's a gap between benevolent-sounding English and actually benevolent algorithms which is the source of difficulty in FAI.  "Just make suggestions - don't do anything!" is, in the end, just more English.

3.  Why we haven't already discussed Holden's suggestion

One of the things that bothers me most about SI is that there is practically no public content, as far as I can tell, explicitly addressing the idea of a "tool" and giving arguments for why AGI is likely to work only as an "agent."

The above statement seems to lack perspective on how many different things various people see as the one obvious solution to Friendly AI. Tool AI wasn't the obvious solution to John McCarthy, I.J. Good, or Marvin Minsky. Today's leading AI textbook, Artificial Intelligence: A Modern Approach - where you can learn all about A* search, by the way - discusses Friendly AI and AI risk for 3.5 pages but doesn't mention tool AI as an obvious solution. For Ray Kurzweil, the obvious solution is merging humans and AIs. For Jurgen Schmidhuber, the obvious solution is AIs that value a certain complicated definition of complexity in their sensory inputs. Ben Goertzel, J. Storrs Hall, and Bill Hibbard, among others, have all written about how silly Singinst is to pursue Friendly AI when the solution is obviously X, for various different X. Among current leading people working on serious AGI programs labeled as such, neither Demis Hassabis (VC-funded to the tune of several million dollars) nor Moshe Looks (head of AGI research at Google) nor Henry Markram (Blue Brain at IBM) think that the obvious answer is Tool AI. Vernor Vinge, Isaac Asimov, and any number of other SF writers with technical backgrounds who spent serious time thinking about these issues didn't converge on that solution.

Obviously I'm not saying that nobody should be allowed to propose solutions because someone else would propose a different solution. I have been known to advocate for particular developmental pathways for Friendly AI myself. But I haven't, for example, told Peter Norvig that deterministic self-modification is such an obvious solution to Friendly AI that I would mistrust his whole AI textbook if he didn't spend time discussing it.

At one point in his conversation with Tallinn, Holden argues that AI will inevitably be developed along planning-Oracle lines, because making suggestions to humans is the natural course that most software takes. Searching for counterexamples instead of positive examples makes it clear that most lines of code don't do this.  Your computer, when it reallocates RAM, doesn't pop up a button asking you if it's okay to reallocate RAM in such-and-such a fashion. Your car doesn't pop up a suggestion when it wants to change the fuel mix or apply dynamic stability control. Factory robots don't operate as human-worn bracelets whose blinking lights suggest motion. High-frequency trading programs execute stock orders on a microsecond timescale. Software that does happen to interface with humans is selectively visible and salient to humans, especially the tiny part of the software that does the interfacing; but this is a special case of a general cost/benefit tradeoff which, more often than not, turns out to swing the other way, because human advice is either too costly or doesn't provide enough benefit. Modern AI programmers are generally more interested in e.g. pushing the technological envelope to allow self-driving cars than to "just" do Google Maps. Branches of AI that invoke human aid, like hybrid chess-playing algorithms designed to incorporate human advice, are a field of study; but they're the exception rather than the rule, and occur primarily where AIs can't yet do something humans do, e.g. humans acting as oracles for theorem-provers, where the humans suggest a route to a proof and the AI actually follows that route. This is another reason why planning Oracles were not a uniquely obvious solution to the various academic AI researchers, would-be AI-creators, SF writers, etcetera, listed above. Again, regardless of whether a planning Oracle is actually the best solution, Holden seems to be empirically-demonstrably overestimating the degree to which other people will automatically have his preferred solution come up first in their search ordering.

4.  Why we should have full-time Friendly AI specialists just like we have trained professionals doing anything else mathy that somebody actually cares about getting right, like pricing interest-rate options or something

I hope that the preceding discussion has made, by example instead of mere argument, what's probably the most important point: If you want to have a sensible discussion about which AI designs are safer, there are specialized skills you can apply to that discussion, as built up over years of study and practice by someone who specializes in answering that sort of question.

This isn't meant as an argument from authority. It's not meant as an attempt to say that only experts should be allowed to contribute to the conversation. But it is meant to say that there is (and ought to be) room in the world for Friendly AI specialists, just like there's room in the world for specialists on optimal philanthropy (e.g. Holden).

The decision to build a non-self-modifying planning Oracle would be properly made by someone who: understood the risk gradient for self-modifying vs. non-self-modifying programs; understood the risk gradient for having the AI thinking about the thought processes of the human watcher and trying to come up with plans implementable by the human watcher in the service of locally absorbed utility functions, vs. trying to implement its own plans in the service of more globally descriptive utility functions; and who, above all, understood on a technical level what exactly gets accomplished by having the plans routed through a human. I've given substantial previous thought to describing more precisely what happens — what is being gained, and how much is being gained — when a human "approves a suggestion" made by an AI. But that would be another a different topic, plus I haven't made too much progress on saying it precisely anyway.

In the transcript of Holden's conversation with Jaan Tallinn, it looked like Tallinn didn't deny the assertion that Friendly AI skills would be inapplicable if we're building a Google Maps AGI. I would deny that assertion and emphasize that denial, because to me it seems that it is exactly Friendly AI programmers who would be able to tell you if the risk gradient for non-self-modification vs. self-modification, the risk gradient for routing plans through humans vs. acting as an agent, the risk gradient for requiring human approval vs. unapproved action, and the actual feasibility of directly constructing transhuman modeling-prediction-and-planning algorithms through directly design of sheerly better computations than are presently run by the human brain, had the right combination of properties to imply that you ought to go construct a non-self-modifying planning Oracle. Similarly if you wanted an AI that took a limited set of actions in the world with human approval, or if you wanted an AI that "just answered questions instead of making plans".

It is similarly implied that a "philosophical AI" might obsolete Friendly AI programmers. If we're talking about PAI that can start with a human's terrible decision theory and come up with a good decision theory, or PAI that can start from a human talking about bad metaethics and then construct a good metaethics... I don't want to say "impossible", because, after all, that's just what human philosophers do. But we are not talking about a trivial invention here. Constructing a "philosophical AI" is a Holy Grail precisely because it's FAI-complete (just ask it "What AI should we build?"), and has been discussed (e.g. with and by Wei Dai) over the years on the old SL4 mailing list and the modern Less Wrong. But it's really not at all clear how you could write an algorithm which would knowably produce the correct answer to the entire puzzle of anthropic reasoning, without being in possession of that correct answer yourself (in the same way that we can have Deep Blue win chess games without knowing the exact moves, but understanding exactly what abstract work Deep Blue is doing to solve the problem).

Holden's post presents a restrictive view of what "Friendly AI" people are supposed to learn and know — that it's about machine learning for optimizing orange sales but not apple sales, or about producing an "agent" that implements CEV — which is something of a straw view, much weaker than the view that a Friendly AI programmer takes of Friendly AI programming. What the human species needs from an x-risk perspective is experts on This Whole Damn Problem, who will acquire whatever skills are needed to that end. The Singularity Institute exists to host such people and enable their research—once we have enough funding to find and recruit them.  See also, How to Purchase AI Risk Reduction.

I'm pretty sure Holden has met people who think that having a whole institute to rate the efficiency of charities is pointless overhead, especially people who think that their own charity-solution is too obviously good to have to contend with busybodies pretending to specialize in thinking about 'marginal utility'.  Which Holden knows about, I would guess, from being paid quite well to think about that economic details when he was a hedge fundie, and learning from books written by professional researchers before then; and the really key point is that people who haven't studied all that stuff don't even realize what they're missing by trying to wing it.  If you don't know, you don't know what you don't know, or the cost of not knowing.  Is there a problem of figuring out who might know something you don't, if Holden insists that there's this strange new stuff called 'marginal utility' you ought to learn about?  Yes, there is.  But is someone who trusts their philanthropic dollars to be steered just by the warm fuzzies of their heart, doing something wrong?  Yes, they are.  It's one thing to say that SIAI isn't known-to-you to be doing it right - another thing still to say that SIAI is known-to-you to be doing it wrong - and then quite another thing entirely to say that there's no need for Friendly AI programmers and you know it, that anyone can see it without resorting to math or cracking a copy of AI: A Modern Approach.  I do wish that Holden would at least credit that the task SIAI is taking on contains at least as many gotchas, relative to the instinctive approach, as optimal philanthropy compared to instinctive philanthropy, and might likewise benefit from some full-time professionally specialized attention, just as our society creates trained professionals to handle any other problem that someone actually cares about getting right.

On the other side of things, Holden says that even if Friendly AI is proven and checked:

"I believe that the probability of an unfavorable outcome - by which I mean an outcome essentially equivalent to what a UFAI would bring about - exceeds 90% in such a scenario."

It's nice that this appreciates that the problem is hard.  Associating all of the difficulty with agenty proposals and thinking that it goes away as soon as you invoke tooliness is, well, of this I've already spoken. I'm not sure whether this irreducible-90%-doom assessment is based on a common straw version of FAI where all the work of the FAI programmer goes into "proving" something and doing this carefully checked proof which then - alas, poor Spock! - turns out to be no more relevant than proving that the underlying CPU does floating-point arithmetic correctly if the transistors work as stated. I've repeatedly said that the idea behind proving determinism of self-modification isn't that this guarantees safety, but that if you prove the self-modification stable the AI might work, whereas if you try to get by with no proofs at all, doom is guaranteed. My mind keeps turning up Ben Goertzel as the one who invented this caricature - "Don't you understand, poor fool Eliezer, life is full of uncertainty, your attempt to flee from it by refuge in 'mathematical proof' is doomed" - but I'm not sure he was actually the inventor. In any case, the burden of safety isn't carried just by the proof, it's carried mostly by proving the right thing. If Holden is assuming that we're just running away from the inherent uncertainty of life by taking refuge in mathematical proof, then, yes, 90% probability of doom is an understatement, the vast majority of plausible-on-first-glance goal criteria you can prove stable will also kill you.

If Holden's assessment does take into account a great effort to select the right theorem to prove - and attempts to incorporate the difficult but finitely difficult feature of meta-level error-detection, as it appears in e.g. the CEV proposal - and he is still assessing 90% doom probability, then I must ask, "What do you think you know and how do you think you know it?" The complexity of the human mind is finite; there's only so many things we want or would-want. Why would someone claim to know that proving the right thing is beyond human ability, even if "100 of the world's most intelligent and relevantly experienced people" (Holden's terms) check it over? There's hidden complexity of wishes, but not infinite complexity of wishes or unlearnable complexity of wishes. There are deep and subtle gotchas but not an unending number of them. And if that were the setting of the hidden variables - how would you end up knowing that with 90% probability in advance? I don't mean to wield my own ignorance as a sword or engage in motivated uncertainty - I hate it when people argue that if they don't know something, nobody else is allowed to know either - so please note that I'm also counterarguing from positive facts pointing the other way: the human brain is complicated but not infinitely complicated, there are hundreds or thousands of cytoarchitecturally distinct brain areas but not trillions or googols.  If humanity had two hundred years to solve FAI using human-level intelligence and there was no penalty for guessing wrong I would be pretty relaxed about the outcome.  If Holden says there's 90% doom probability left over no matter what sane intelligent people do (all of which goes away if you just build Google Maps AGI, but leave that aside for now) I would ask him what he knows now, in advance, that all those sane intelligent people will miss.  I don't see how you could (well-justifiedly) access that epistemic state.

I acknowledge that there are points in Holden's post which are not addressed in this reply, acknowledge that these points are also deserving of reply, and hope that other SIAI personnel will be able to reply to them.

Comments (340)

Comment author: John_Maxwell_IV 12 June 2012 04:14:21AM *  48 points [-]

My summary (now with endorsement by Eliezer!):

  • SI can be a valuable organization even if Tool AI turns out to be the right approach:
    • Skills/organizational capabilities for safe Tool AI are similar to those for Friendly AI.
    • EY seems to imply that much of SI's existing body of work can be reused.
    • Offhand remark that seemed important: Superintelligent Tool AI would be more difficult since it would have to be developed in way that it would not recursively self-improve.
  • Tool AI is nontrivial:
    • The number of possible plans is way too large for an AI to realistically evaluate all them. Heuristics will have to be used to find suboptimal but promising plans.
    • The reasoning behind the plan the AI chooses might be way beyond the comprehension of the user. It's not clear how best to deal with this, given that the AI is only approximating the user's wishes and can't really be trusted to choose plans without supervision.
    • Constructing a halfway decent approximation of the user's utility function and having a model good enough to make plans with are also far from solved problems.
    • Potential Tool AI gotcha: The AI might give you a self-fulfilling negative prophecy that the AI didn't realize would harm you.
    • These are just examples. Point is, saying "but the AI will just do this!" is far removed from specifying the AI in a rigorous formal way and proving it will do that.
  • Tool AI is not obviously the way AGI should or will be developed:
    • Many leading AGI thinkers have their own pet idea about what AGI should do. Few to none endorse Tool AI. If it was obvious all the leading AGI thinkers would endorse it.
    • Actually, most modern AI applications don't involve human input, so it's not obvious that AGI will develop along Tool AI lines.
  • Full-time Friendliness researchers are worth having:
    • If nothing else, they're useful for evaluating proposals like Holden's Tool AI one to figure out if they are really sound.
    • Friendliness philosophy would be difficult to program an AI to do. Even if we thought we had a program that could do it, how would we know the answers from that program were correct? So we probably need humans.
    • Friendliness researchers need to have a broader domain of expertise than Holden gives them credit for. They need to have expertise in whatever happens to be necessary to ensure safe AI.
    • The problems of Friendliness are tricky, so laypeople should beware of jumping to conclusions about Friendliness.
  • Holden's estimate of a 90% chance of doom even given a 100 person FAI team approving the design is overly pessimistic:
    • EY is aware it's extremely difficult to know what properties about a prospective FAI need to be formally proved, and plans to put a lot of effort into figuring this out.
    • The difficulty of Friendliness is finite. The difficulties are big and subtle, but not unending.
    • Where did 90% come from? Lots of uncertainty here...
  • Holden made other good points not addressed here.
Comment author: ChristianKl 17 June 2012 10:08:46PM 9 points [-]

The difficulty of Friendliness is finite. The difficulties are big and subtle, but not unending.

How do we know that the problem is finite? When it comes to proving a computer program safe from being hacked the problem is considered NP-hard. Google Chrome got recently hacked by chaining 14 different bugs together. A working AGI is probably as least a complex as Google Chrome. Proving it safe will likely also be NP-hard.

Google Chrome doesn't even self modify.

Comment author: CarlShulman 12 June 2012 05:50:12AM 9 points [-]

This point seems missing:

You can't get a 20-move solution out of a human brain, using the native human planning algorithm. Humanity can do it, but only by exploiting the ability of humans to explicitly comprehend the deep structure of the domain (not just rely on intuition) and then inventing an artifact, a new design, running code which uses a different and superior cognitive algorithm, to solve that Rubik's Cube in 20 moves. We do all that without being self-modifying, but it's still a capability to respect.

A system that undertakes extended processes of research and thinking, generating new ideas and writing new programs for internal experiments, seems both much more effective and much more potentially risky than something like chess program with a simple fixed algorithm to search using a fixed narrow representation of the world (as a chess board).

Comment author: Eliezer_Yudkowsky 12 June 2012 05:53:02AM 10 points [-]

Looks pretty good, actually. Nice.

Comment author: SilasBarta 23 June 2012 04:49:42AM 2 points [-]

So you wrote 10x too much then?

Comment author: beoShaffer 12 June 2012 04:45:37AM 1 point [-]

That is about how I read it.

Comment author: HoldenKarnofsky 05 July 2012 04:18:16PM 17 points [-]

Hello,

I appreciate the thoughtful response. I plan to respond at greater length in the future, both to this post and to some other content posted by SI representatives and commenters. For now, I wanted to take a shot at clarifying the discussion of "tool-AI" by discussing AIXI. One of the the issues I've found with the debate over FAI in general is that I haven't seen much in the way of formal precision about the challenge of Friendliness (I recognize that I have also provided little formal precision, though I feel the burden of formalization is on SI here). It occurred to me that AIXI might provide a good opportunity to have a more precise discussion, if in fact it is believed to represent a case of "a rare exception who specified his AGI in such unambiguous mathematical terms that he actually succeeded at realizing, after some discussion with SIAI personnel, that AIXI would kill off its users and seize control of its reward button."

So here's my characterization of how one might work toward a safe and useful version of AIXI, using the "tool-AI" framework, if one could in fact develop an efficient enough approximation of AIXI to qualify as a powerful AGI. Of course, this is just a rough outline of what I have in mind, but hopefully it adds some clarity to the discussion.

A. Write a program that

  1. Computes an optimal policy, using some implementation of equation (20) on page 22 of http://www.hutter1.net/ai/aixigentle.pdf
  2. "Prints" the policy in a human-readable format (using some fixed algorithm for "printing" that is not driven by a utility function)
  3. Provides tools for answering user questions about the policy, i.e., "What will be its effect on _?" (using some fixed algorithm for answering user questions that makes use of AIXI's probability function, and is not driven by a utility function)
  4. Does not contain any procedures for "implementing" the policy, only for displaying it and its implications in human-readable form

B. Run the program; examine its output using the tools described above (#2 and #3); if, upon such examination, the policy appears potentially destructive, continue tweaking the program (for example, by tweaking the utility it is selecting a policy to maximize) until the policy appears safe and desirable

C. Implement the policy using tools other than AIXI agent

D. Repeat (B) and (C) until one has confidence that the AIXI agent reliably produces safe and desirable policies, at which point more automation may be called for

My claim is that this approach would be superior to that of trying to develop "Friendliness theory" in advance of having any working AGI, because it would allow experiment- rather than theory-based development. Eliezer, I'm interested in your thoughts about my claim. Do you agree? If not, where is our disagreement?

Comment author: Eliezer_Yudkowsky 11 July 2012 10:59:07PM 27 points [-]

Didn't see this at the time, sorry.

So... I'm sorry if this reply seems a little unhelpful, and I wish there was some way to engage more strongly, but...

Point (1) is the main problem. AIXI updates freely over a gigantic range of sensory predictors with no specified ontology - it's a sum over a huge set of programs, and we, the users, have no idea what the representations are talking about, except that at the end of their computations they predict, "You will see a sensory 1 (or a sensory 0)." (In my preferred formalism, the program puts a probability on a 0 instead.) Inside, the program could've been modeling the universe in terms of atoms, quarks, quantum fields, cellular automata, giant moving paperclips, slave agents scurrying around... we, the programmers, have no idea how AIXI is modeling the world and producing its predictions, and indeed, the final prediction could be a sum over many different representations.

This means that equation (20) in Hutter is written as a utility function over sense data, where the reward channel is just a special case of sense data. We can easily adapt this equation to talk about any function computed directly over sense data - we can get AIXI to optimize any aspect of its sense data that we please. We can't get it to optimize a quality of the external universe. One of the challenges I listed in my FAI Open Problems talk, and one of the problems I intend to talk about in my FAI Open Problems sequence, is to take the first nontrivial steps toward adapting this formalism - to e.g. take an equivalent of AIXI in a really simple universe, with a really simple goal, something along the lines of a Life universe and a goal of making gliders, and specify something given unlimited computing power which would behave like it had that goal, without pre-fixing the ontology of the causal representation to that of the real universe, i.e., you want something that can range freely over ontologies in its predictive algorithms, but which still behaves like it's maximizing an outside thing like gliders instead of a sensory channel like the reward channel. This is an unsolved problem!

We haven't even got to the part where it's difficult to say in formal terms how to interpret what a human says s/he wants the AI to plan, and where failures of phrasing of that utility function can also cause a superhuman intelligence to kill you. We haven't even got to the huge buried FAI problem inside the word "optimal" in point (1), which is the really difficult part in the whole thing. Because so far we're dealing with a formalism that can't even represent a purpose of the type you're looking for - it can only optimize over sense data, and this is not a coincidental fact, but rather a deep problem which the AIXI formalism deliberately avoided.

(2) sounds like you think an AI with an alien, superhuman planning algorithm can tell humans what to do without ever thinking consequentialistically about which different statements will result in human understanding or misunderstanding. Anna says that I need to work harder on not assuming other people are thinking silly things, but even so, when I look at this, it's hard not to imagine that you're modeling AIXI as a sort of spirit containing thoughts, whose thoughts could be exposed to the outside with a simple exposure-function. It's not unthinkable that a non-self-modifying superhuman planning Oracle could be developed with the further constraint that its thoughts are human-interpretable, or can be translated for human use without any algorithms that reason internally about what humans understand, but this would at the least be hard. And with AIXI it would be impossible, because AIXI's model of the world ranges over literally all possible ontologies and representations, and its plans are naked motor outputs.

Similar remarks apply to interpreting and answering "What will be its effect on _?" It turns out that getting an AI to understand human language is a very hard problem, and it may very well be that even though talking doesn't feel like having a utility function, our brains are using consequential reasoning to do it. Certainly, when I write language, that feels like I'm being deliberate. It's also worth noting that "What is the effect on X?" really means "What are the effects I care about on X?" and that there's a large understanding-the-human's-utility-function problem here. In particular, you don't want your language for describing "effects" to partition, as the same state of described affairs, any two states which humans assign widely different utilities. Let's say there are two plans for getting my grandmother out of a burning house, one of which destroys her music collection, one of which leaves it intact. Does the AI know that music is valuable? If not, will it not describe music-destruction as an "effect" of a plan which offers to free up large amounts of computer storage by, as it turns out, overwriting everyone's music collection? If you then say that the AI should describe changes to files in general, well, should it also talk about changes to its own internal files? Every action comes with a huge number of consequences - if we hear about all of them (reality described on a level so granular that it automatically captures all utility shifts, as well as a huge number of other unimportant things) then we'll be there forever.

I wish I had something more cooperative to say in reply - it feels like I'm committing some variant of logical rudeness by this reply - but the truth is, it seems to me that AIXI isn't a good basis for the agent you want to describe; and I don't know how to describe it formally myself, either.

Comment author: HoldenKarnofsky 18 July 2012 02:35:33AM 16 points [-]

Thanks for the response. To clarify, I'm not trying to point to the AIXI framework as a promising path; I'm trying to take advantage of the unusually high degree of formalization here in order to gain clarity on the feasibility and potential danger points of the "tool AI" approach.

It sounds to me like your two major issues with the framework I presented are (to summarize):

(1) There is a sense in which AIXI predictions must be reducible to predictions about the limited set of inputs it can "observe directly" (what you call its "sense data").

(2) Computers model the world in ways that can be unrecognizable to humans; it may be difficult to create interfaces that allow humans to understand the implicit assumptions and predictions in their models.

I don't claim that these problems are trivial to deal with. And stated as you state them, they sound abstractly very difficult to deal with. However, it seems true - and worth noting - that "normal" software development has repeatedly dealt with them successfully. For example: Google Maps works with a limited set of inputs; Google Maps does not "think" like I do and I would not be able to look at a dump of its calculations and have any real sense for what it is doing; yet Google Maps does make intelligent predictions about the external universe (e.g., "following direction set X will get you from point A to point B in reasonable time"), and it also provides an interface (the "route map") that helps me understand its predictions and the implicit reasoning (e.g. "how, why, and with what other consequences direction set X will get me from point A to point B").

Difficult though it may be to overcome these challenges, my impression is that software developers have consistently - and successfully - chosen to take them on, building algorithms that can be "understood" via interfaces and iterated over - rather than trying to prove the safety and usefulness of their algorithms with pure theory before ever running them. Not only does the former method seem "safer" (in the sense that it is less likely to lead to putting software in production before its safety and usefulness has been established) but it seems a faster path to development as well.

It seems that you see a fundamental disconnect between how software development has traditionally worked and how it will have to work in order to result in AGI. But I don't understand your view of this disconnect well enough to see why it would lead to a discontinuation of the phenomenon I describe above. In short, traditional software development seems to have an easier (and faster and safer) time overcoming the challenges of the "tool" framework than overcoming the challenges of up-front theoretical proofs of safety/usefulness; why should we expect this to reverse in the case of AGI?

Comment author: Eliezer_Yudkowsky 18 July 2012 02:12:12PM 25 points [-]

So first a quick note: I wasn't trying to say that the difficulties of AIXI are universal and everything goes analogously to AIXI, I was just stating why AIXI couldn't represent the suggestion you were trying to make. The general lesson to be learned is not that everything else works like AIXI, but that you need to look a lot harder at an equation before thinking that it does what you want.

On a procedural level, I worry a bit that the discussion is trying to proceed by analogy to Google Maps. Let it first be noted that Google Maps simply is not playing in the same league as, say, the human brain, in terms of complexity; and that if we were to look at the winning "algorithm" of the million-dollar Netflix Prize competition, which was in fact a blend of 107 different algorithms, you would have a considerably harder time figuring out why it claimed anything it claimed.

But to return to the meta-point, I worry about conversations that go into "But X is like Y, which does Z, so X should do reinterpreted-Z". Usually, in my experience, that goes into what I call "reference class tennis" or "I'm taking my reference class and going home". The trouble is that there's an unlimited number of possible analogies and reference classes, and everyone has a different one. I was just browsing old LW posts today (to find a URL of a quick summary of why group-selection arguments don't work in mammals) and ran across a quotation from Perry Metzger to the effect that so long as the laws of physics apply, there will always be evolution, hence nature red in tooth and claw will continue into the future - to him, the obvious analogy for the advent of AI was "nature red in tooth and claw", and people who see things this way tend to want to cling to that analogy even if you delve into some basic evolutionary biology with math to show how much it isn't like intelligent design. For Robin Hanson, the one true analogy is to the industrial revolution and farming revolutions, meaning that there will be lots of AIs in a highly competitive economic situation with standards of living tending toward the bare minimum, and this is so absolutely inevitable and consonant with The Way Things Should Be as to not be worth fighting at all. That's his one true analogy and I've never been able to persuade him otherwise. For Kurzweil, the fact that many different things proceed at a Moore's Law rate to the benefit of humanity means that all these things are destined to continue and converge into the future, also to the benefit of humanity. For him, "things that go by Moore's Law" is his favorite reference class.

I can have a back-and-forth conversation with Nick Bostrom, who looks much more favorably on Oracle AI in general than I do, because we're not playing reference class tennis with "But surely that will be just like all the previous X-in-my-favorite-reference-class", nor saying, "But surely this is the inevitable trend of technology"; instead we lay out particular, "Suppose we do this?" and try to discuss how it will work, not with any added language about how surely anyone will do it that way, or how it's got to be like Z because all previous Y were like Z, etcetera.

My own FAI development plans call for trying to maintain programmer-understandability of some parts of the AI during development. I expect this to be a huge headache, possibly 30% of total headache, possibly the critical point on which my plans fail, because it doesn't happen naturally. Go look at the source code of the human brain and try to figure out what a gene does. Go ask the Netflix Prize winner for a movie recommendation and try to figure out "why" it thinks you'll like watching it. Go train a neural network and then ask why it classified something as positive or negative. Try to keep track of all the memory allocations inside your operating system - that part is humanly understandable, but it flies past so fast you can only monitor a tiny fraction of what goes on, and if you want to look at just the most "significant" parts, you would need an automated algorithm to tell you what's significant. Most AI algorithms are not humanly understandable. Part of Bayesianism's appeal in AI is that Bayesian programs tend to be more understandable than non-Bayesian AI algorithms. I have hopeful plans to try and constrain early FAI content to humanly comprehensible ontologies, prefer algorithms with humanly comprehensible reasons-for-outputs, carefully weigh up which parts of the AI can safely be less comprehensible, monitor significant events, slow down the AI so that this monitoring can occur, and so on. That's all Friendly AI stuff, and I'm talking about it because I'm an FAI guy. I don't think I've ever heard any other AGI project express such plans; and in mainstream AI, human-comprehensibility is considered a nice feature, but rarely a necessary one.

It should finally be noted that AI famously does not result from generalizing normal software development. If you start with a map-route program and then try to program it to plan more and more things until it becomes an AI... you're doomed, and all the experienced people know you're doomed. I think there's an entry or two in the old Jargon File aka Hacker's Dictionary to this effect. There's a qualitative jump to writing a different sort of software - from normal programming where you create a program conjugate to the problem you're trying to solve, to AI where you try to solve cognitive-science problems so the AI can solve the object-level problem. I've personally met a programmer or two who've generalized their code in interesting ways, and who feel like they ought to be able to generalize it even further until it becomes intelligent. This is a famous illusion among aspiring young brilliant hackers who haven't studied AI. Machine learning is a separate discipline and involves algorithms and problems that look quite different from "normal" programming.

Comment author: HoldenKarnofsky 18 July 2012 04:29:00PM 14 points [-]

Thanks for the response. My thoughts at this point are that

  • We seem to have differing views of how to best do what you call "reference class tennis" and how useful it can be. I'll probably be writing about my views more in the future.
  • I find it plausible that AGI will have to follow a substantially different approach from "normal" software. But I'm not clear on the specifics of what SI believes those differences will be and why they point to the "proving safety/usefulness before running" approach over the "tool" approach.
  • We seem to have differing views of how frequently today's software can be made comprehensible via interfaces. For example, my intuition is that the people who worked on the Netflix Prize algorithm had good interfaces for understanding "why" it recommends what it does, and used these to refine it. I may further investigate this matter (casually, not as a high priority); on SI's end, it might be helpful (from my perspective) to provide detailed examples of existing algorithms for which the "tool" approach to development didn't work and something closer to "proving safety/usefulness up front" was necessary.
Comment author: oooo 06 July 2013 05:50:31PM *  3 points [-]

Canonical software development examples emphasizing "proving safety/usefulness before running" over the "tool" software development approach are cryptographic libraries and NASA space shuttle navigation.

At the time of writing this comment, there was recent furor over software called CryptoCat that didn't provide enough warnings that it was not properly vetted by cryptographers and thus should have been assumed to be inherently insecure. Conventional wisdom and repeated warnings from the security community state that cryptography is extremely difficult to do properly and attempting to create your own may result in catastrophic results. A similar thought and development process goes into space shuttle code.

It seems that the FAI approach to "proving safety/usefulness" is more similar to the way cryptographic algorithms are developed than the (seemingly) much faster "tool" approach, which is more akin to web development where the stakes aren't quite as high.

EDIT: I believe the "prove" approach still allows one to run snippets of code in isolation, but tends to shy away from running everything end-to-end until significant effort has gone into individual component testing.

Comment author: wedrifid 19 July 2012 02:05:49AM 1 point [-]

I can have a back-and-forth conversation with Nick Bostrom, who looks much more favorably on Oracle AI in general than I do, because we're not playing reference class tennis with "But surely that will be just like all the previous X-in-my-favorite-reference-class", nor saying, "But surely this is the inevitable trend of technology"; instead we lay out particular, "Suppose we do this?" and try to discuss how it will work, not with any added language about how surely anyone will do it that way, or how it's got to be like Z because all previous Y were like Z, etcetera.

That's one way to "win" a game of reference class tennis. Declare unilaterally that what you are discussing falls into the reference class "things that are most effectively reasoned about by discussing low level details and abandoning or ignoring all observed evidence about how things with various kinds of similarity have worked in the past". Sure, it may lead to terrible predictions sometimes but by golly, it means you can score an 'ace' in the reference class tennis while pretending you are not even playing!

Comment author: Eliezer_Yudkowsky 19 July 2012 05:52:40PM 9 points [-]

And atheism is a religion, and bald is a hair color.

The three distinguishing characteristics of "reference class tennis" are (1) that there are many possible reference classes you could pick and everyone engaging in the tennis game has their own favorite which is different from everyone else's; (2) that the actual thing is obviously more dissimilar to all the cited previous elements of the so-called reference class than all those elements are similar to each other (if they even form a natural category at all rather than having being picked out retrospectively based on similarity of outcome to the preferred conclusion); and (3) that the citer of the reference class says it with a cognitive-traffic-signal quality which attempts to shut down any attempt to counterargue the analogy because "it always happens like that" or because we have so many alleged "examples" of the "same outcome" occurring (for Hansonian rationalists this is accompanied by a claim that what you are doing is the "outside view" (see point 2 and 1 for why it's not) and that it would be bad rationality to think about the "individual details").

I have also termed this Argument by Greek Analogy after Socrates's attempt to argue that, since the Sun appears the next day after setting, souls must be immortal.

Comment author: [deleted] 19 July 2012 10:20:53PM *  11 points [-]

I have also termed this Argument by Greek Analogy after Socrates's attempt to argue that, since the Sun appears the next day after setting, souls must be immortal.

For the curious, this is from the Phaedo pages 70-72. The run of the argument are basically thus:

P1 Natural changes are changes from and to opposites, like hot from relatively cold, etc.

P2 Since every change is between opposites A and B, there are two logically possible processes of change, namely A to B and B to A.

P3 If only one of the two processes were physically possible, then we should expect to see only one of the two opposites in nature, since the other will have passed away irretrievably.

P4 Life and death are opposites.

P5 We have experience of the process of death.

P6 We have experience of things which are alive

C From P3, 4, 5, and 6 there is a physically possible, and actual, process of going from death to life.

The argument doesn't itself prove (haha) the immortality of the soul, only that living things come from dead things. The argument is made in support of the claim, made prior to this argument, that if living people come from dead people, then dead people must exist somewhere. The argument is particularly interesting for premises 1 and 2, which are hard to deny, and 3, which seems fallacious but for non-obvious reasons.

Comment author: Eliezer_Yudkowsky 20 July 2012 04:40:00PM 5 points [-]

This sounds like it might be a bit of a reverent-Western-scholar steelman such as might be taught in modern philosophy classes; Plato's original argument for the immortality of the soul sounded more like this, which is why I use it as an early exemplar of reference class tennis:

-

Then let us consider the whole question, not in relation to man only, but in relation to animals generally, and to plants, and to everything of which there is generation, and the proof will be easier. Are not all things which have opposites generated out of their opposites? I mean such things as good and evil, just and unjust—and there are innumerable other opposites which are generated out of opposites. And I want to show that in all opposites there is of necessity a similar alternation; I mean to say, for example, that anything which becomes greater must become greater after being less.

True.

And that which becomes less must have been once greater and then have become less.

Yes.

And the weaker is generated from the stronger, and the swifter from the slower.

Very true.

And the worse is from the better, and the more just is from the more unjust.

Of course.

And is this true of all opposites? and are we convinced that all of them are generated out of opposites?

Yes.

And in this universal opposition of all things, are there not also two intermediate processes which are ever going on, from one to the other opposite, and back again; where there is a greater and a less there is also an intermediate process of increase and diminution, and that which grows is said to wax, and that which decays to wane?

Yes, he said.

And there are many other processes, such as division and composition, cooling and heating, which equally involve a passage into and out of one another. And this necessarily holds of all opposites, even though not always expressed in words—they are really generated out of one another, and there is a passing or process from one to the other of them?

Very true, he replied.

Well, and is there not an opposite of life, as sleep is the opposite of waking?

True, he said.

And what is it?

Death, he answered.

And these, if they are opposites, are generated the one from the other, and have there their two intermediate processes also?

Of course.

Now, said Socrates, I will analyze one of the two pairs of opposites which I have mentioned to you, and also its intermediate processes, and you shall analyze the other to me. One of them I term sleep, the other waking. The state of sleep is opposed to the state of waking, and out of sleeping waking is generated, and out of waking, sleeping; and the process of generation is in the one case falling asleep, and in the other waking up. Do you agree?

I entirely agree.

Then, suppose that you analyze life and death to me in the same manner. Is not death opposed to life?

Yes.

And they are generated one from the other?

Yes.

What is generated from the living?

The dead.

And what from the dead?

I can only say in answer—the living.

Then the living, whether things or persons, Cebes, are generated from the dead?

That is clear, he replied.

Then the inference is that our souls exist in the world below?

That is true.

(etc.)

Comment author: [deleted] 20 July 2012 07:36:25PM 2 points [-]

This sounds like it might be a bit of a reverent-Western-scholar steelman such as might be taught in modern philosophy classes

That was roughly my aim, but I don't think I inserted any premises that weren't there. Did you have a complaint about the accuracy of my paraphrase? The really implausible premise there, namely that death is the opposite of life, is preserved I think.

As for reverence, why not? He was, after all, the very first person in our historical record to suggest that thinking better might make you happier. He was also an intellectualist about morality, at least sometimes a hedonic utilitarian, and held no great respect for logic. And he was a skilled myth-maker. He sounds like a man after your own heart, actually.

Comment author: thomblake 25 July 2012 08:14:40PM 2 points [-]

I think your summary didn't leave anything out, or even apply anything particularly charitable.

Comment author: thomblake 25 July 2012 08:18:57PM 0 points [-]

Esar's summary doesn't seem to be different from this, other than 1) adding the useful bit about "passed away irretrievably" and 2) yours makes it clear that the logical jump happens right at the end.

I'm actually not sure now why you consider this like "reference class tennis". The argument looks fine, except for the part where "souls exist in the world below" jumps in as a conclusion, not having been mentioned earlier in the argument.

Comment author: [deleted] 25 July 2012 08:50:02PM *  0 points [-]

The 'souls exist in the world below' bit is directly before what Eliezer quoted:

Suppose we consider the question whether the souls of men after death are or are not in the world below. There comes into my mind an ancient doctrine which affirms that they go from hence into the other world, and returning hither, are born again from the dead. Now if it be true that the living come from the dead, then our souls must exist in the other world, for if not, how could they have been born again? And this would be conclusive, if there were any real evidence that the living are only born from the dead; but if this is not so, then other arguments will have to be adduced.

Very true, replied Cebes.

Then let us consider the whole question...

But you're right that nothing in the argument defends the idea of a world below, just that souls must exist in some way between bodies.

Comment author: wedrifid 20 July 2012 12:15:37AM 5 points [-]

(2) that the actual thing is obviously more dissimilar to all the cited previous elements of the so-called reference class than all those elements are similar to each other (if they even form a natural category at all rather than having being picked out retrospectively based on similarity of outcome to the preferred conclusion);

Ok, it seems like under this definition of "reference class tennis" (particularly parts (2) and (3)) the participants must be wrong and behaving irrationality about it in order to be playing reference class tennis. So when they are either right or at least applying "outside view" considerations correctly, given all the information available to them they aren't actually playing "reference class tennis" but instead doing whatever it is that reasoning (boundedly) correctly using reference to actual relevant evidence about related occurrences is called when it isn't packaged with irrational wrongness.

With this definition in mind it is necessary to translate replies such as those here by Holden:

We seem to have differing views of how to best do what you call "reference class tennis" and how useful it can be. I'll probably be writing about my views more in the future.

Holden's meaning is, of course, not that that he argues <reference class tennis: (1), (2), (3)> is actually a good thing but rather declaring that the label doesn't apply to what he is doing. He is instead doing that other thing that is actually sound thinking and thinks people are correct to do so.

Come to think of it if most people in Holden's shoes heard Eliezer accuse them of "reference class tennis" and actually knew that he intended it with the meaning he explicitly defines here rather than the one they infer from context they would probably just consider him arrogant, rude and mind killed then write him and his organisation off as not worth engaging with.

In the vast majority of cases where I have previously seen Eliezer argue against people using "outside view" I have agreed with Eliezer, and have grown rather fond of using the phrase "reference class tennis" as a reply myself where appropriate. But seeing how far Eliezer has taken the anti-outside-view position here and the extent to which "reference class tennis" is defined as purely an anti-outside-view semantic stop sign I'll be far more hesitant to make us of it myself.

It is tempting to observe "Eliezer is almost always right when he argues against 'outside view' applications, and the other people are all confused. He is currently arguing against 'outside view' applications. Therefore, the other people are probably confused." To that I reply either "Reference class tennis!" or "F*$% you, I'm right and you're wrong!" (I'm honestly not sure which is the least offensive.)

Comment author: Eliezer_Yudkowsky 20 July 2012 12:43:07AM 5 points [-]

Which of 1, 2 and 3 do you disagree with in this case?

Edit: I mean, I'm sorry to parody but I don't really want to carefully rehash the entire thing, so, from my perspective, Holden just said, "But surely strong AI will fall into the reference class of technology used to give users advice, just like Google Maps doesn't drive your car; this is where all technology tends to go, so I'm really skeptical about discussing any other possibility." Only Holden has argued to SI that strong AI falls into this particular reference class so far as I can recall, with many other people having their own favored reference classes e.g. Hanson et. al as cited above; a strong AI is far more internally dissimilar from Google Maps and Yelp than Google Maps and Yelp are internally similar to each other, plus there are many many other software programs that don't provide advice at all so arguably the whole class may be chosen-post-facto; and I'd have to look up Holden's exact words and replies to e.g. Jaan Tallinn to decide to what degree, if any, he used the analogy to foreclose other possibilities conversationally without further debate, but I do think it happened a little, but less so and less explicitly than in my Robin Hanson debate. If you don't think I should at this point diverge into explaining the concept of "reference class tennis", how should the conversation proceed further?

Also, further opinions desired on whether I was being rude, whether logically rude or otherwise.

Comment author: Randaly 25 July 2012 04:39:40AM 7 points [-]

Viewed charitably, you were not being rude, although you did veer away from your main point in ways likely to be unproductive. (For example, being unnecessarily dismissive towards Hanson, who you'd previously stated had given arguments roughly as good as Holden's; or spending so much of your final paragraph emphasizing Holden's lack of knowledge regarding AI.)

On the most likely viewing, it looks like you thought Holden was probably playing reference class tennis. This would have been rude, because it would imply that you thought the following inaccurate things about him:

  • He was "taking his reference class and going home"
  • That you can't "have a back-and-forth conversation" with him

I don't think that you intended those implications. All the same, your final comment came across as noticeably less well-written than your post.

Comment author: Eliezer_Yudkowsky 25 July 2012 05:53:38PM 1 point [-]

Thanks for the third-party opinion!

Comment author: TimS 20 July 2012 12:47:06AM 1 point [-]

I'm confused how you thought "reference class tennis" was anything but a slur on the other side's argument. Likewise "mindkilled." Sometimes, slurs about arguments are justified (agnostic in the instant case) - but that's a separate issue.

Comment author: Sewing-Machine 19 July 2012 10:40:08PM 1 point [-]

The three distinguishing characteristics of "reference class tennis" are

Do Karnofsky's contributions have even one of these characteristics, let alone all of them?

Comment author: Eliezer_Yudkowsky 20 July 2012 12:09:48AM 2 points [-]

Empirically obviously 1 is true, I would argue strongly for 2 but it's a legitimate point of dispute, and I would say that there were relatively small but still noticeable but quite forgiveable traces of 3.

Comment author: JGWeissman 18 July 2012 05:42:17PM 4 points [-]

For example: Google Maps works with a limited set of inputs; Google Maps does not "think" like I do and I would not be able to look at a dump of its calculations and have any real sense for what it is doing; yet Google Maps does make intelligent predictions about the external universe (e.g., "following direction set X will get you from point A to point B in reasonable time"), and it also provides an interface (the "route map") that helps me understand its predictions and the implicit reasoning (e.g. "how, why, and with what other consequences direction set X will get me from point A to point B").

Explaining routes is domain specific and quite simple. When you are using domain specific techniques to find solutions to domain specific problems, you can use domain specific interfaces where human programmers and designers do all the heavy lifting to figure out the general strategy of how to communicate to the user.

But if you want a tool AGI that finds solutions in arbitrary domains, you need a cross domain solution for communicating tool AGI's plans to the user. This is as much a harder problem than showing a route on a map, as cross domain AGI is a harder problem than computing the routes. Instead of the programmer figuring out how to plot road tracing curves on a map, the programmer has to figure out how to get the computer to figure out that displaying a map with route traced over it is a useful thing to do, in a way that generalizes figuring out other useful things to do to communicate answers to other types of questions. And among the hard subproblems of programming computers to find useful things to do in general problems is specifying the meaning of "useful". If that is done poorly, the tool AGI tries to trick the user into accepting plans that achieve some value negating distortion of what we actually want, instead of giving information that helps provide a good evaluation. Doing this right requires solving the same problems required to do FAI right.

Comment author: private_messaging 18 July 2012 06:11:02AM *  0 points [-]

To note something on making AIXI based tool: Instead of calculating rewards sum over the whole future (something that is simultaneously impractical, computationally expensive, and would only serve to impair performance on task at hand), one could use the single-step reward, with 1 for button being pressed any time and 0 for button not being pressed ever. It is still not entirely a tool, but it has very bounded range of unintended behaviour (much harder to speculate of the terminator scenario). In the Hutter's paper he outlines several not-quite-intelligences before arriving at AIXI.

[edit2: also I do not believe that even with the large sum a really powerful AIXI-tl would be intelligently dangerous rather than simply clever at breaking the hardware that's computing it. All the valid models in AIXI-tl that affect the choice of actions have to magically insert actions being probed into some kind of internal world model. The hardware that actually makes those actions, complete with sensory apparatus, is incidental; a useless power drain; a needless fire hazard endangering the precious reward pathway]

With regards to utility functions, the utility functions in the AI sense are real valued functions taken over the world model, not functions like number of paperclips in the world. The latter function, unsafe or safe, would be incredibly difficult or impossible to define using conventional methods. It would suffice for accelerating the progress to have an algorithm that can take in an arbitrary function and find it's maximum; while it would indeed seem to be "very difficult" to use that to cure cancer, it could be plugged into existing models and very quickly be used to e.g. design cellular machinery that would keep repairing the DNA alterations.

Likewise, the speculative tool that can understand phrase 'how to cure cancer' and phrase 'what is the curing time of epoxy' would have to pick up most narrow least objectionable interpretation of the 'cure cancer' phrase to merely answer something more useful than 'cancer is not a type of epoxy or glue, it does not cure'; it seems that not seeing killing everyone as a valid interpretation comes in as necessary consequence of ability to process language at all.

Comment author: CarlShulman 22 July 2012 07:07:04AM 0 points [-]

All the valid models in AIXI-tl that affect the choice of actions have to magically insert actions being probed into some kind of internal world model. The hardware that actually makes those actions, complete with sensory apparatus, is incidental; a useless power drain; a needless fire hazard endangering the precious reward pathway

If the past sensory data include information about the internal workings, then there will be a striking correlation between the outputs that the workings would produce on their own (for physical reasons) and the AI's outputs. That rules out (or drives down expected utility of acting upon) all but very crazy hypotheses about how the Cartesian interaction works. Wrecking the hardware would break that correlation, and it's not clear what the crazy hypotheses would say about that, e.g. hypotheses that some simply specified intelligence is stage-managing the inputs, or that sometimes the AIXI-tl's outputs matter, and other times only the physical hardware matters.

Comment author: cousin_it 13 July 2012 01:54:01PM *  2 points [-]

take an equivalent of AIXI in a really simple universe, with a really simple goal, something along the lines of a Life universe and a goal of making gliders, and specify something given unlimited computing power which would behave like it had that goal, without pre-fixing the ontology of the causal representation to that of the real universe, i.e., you want something that can range freely over ontologies in its predictive algorithms, but which still behaves like it's maximizing an outside thing like gliders instead of a sensory channel like the reward channel.

Your question seems to be about how sentient beings in a Game of Life universe are supposed to define "gliders" to the AI.

1) If they know the true laws of their cellular automaton, they can make a UDT-ish AI that examines statements like "if this logical algorithm has such-and-such output, then my prior over starting configurations of the universe logically implies such-and-such total number of gliders".

2) If they only know that their universe is some cellular automaton and have a prior over all possible automata, they can similarly say "maximize the number of smallest possible spaceships under the automaton rules" and give the AI some sensory channel wide enough to pin down the specific automaton with high probability.

3) If they only know what sensory experiences correspond to the existence of gliders, but don't know what gliders are... I guess we have a problem because sensory experiences can be influenced by the AI :-(

Comment author: TheOtherDave 13 July 2012 02:42:28PM 1 point [-]

Regarding #3: what happens given a directive like "Over there are a bunch of people who report sensory experiences of the kind I'm interested in. Figure out what differentially caused those experiences, and maximize the incidence of that."?

(I'm not concerned with the specifics of my wording, which undoubtedly contains infinite loopholes; I'm asking about the general strategy of, when all I know is sensory experiences, referring to the differential causes of those experiences, whatever they may be. Which, yes, I would expect to include, in the case where there actually are no gliders and the recurring perception of gliders is the result of a glitch in my perceptual system, modifying my perceptual system to make such glitches more likely... but which I would not expect to include, in the case where my perceptual system is operating essentially the same way when it perceives gliders as when it perceives everything else, modifying my perceptual system to include such glitches (since such a glitch is not the differential cause of experiences of gliders in the first place.))

Comment author: cousin_it 13 July 2012 04:42:57PM *  1 point [-]

Let's say you want the AI to maximize the amount of hydrogen, and you formulate the goal as "maximize the amount of the substance most likely referred to by such-and-such state of mind", where "referred to" is cashed out however you like. Now imagine that some other substance is 10x cheaper to make than hydrogen. Then the AI could create a bunch of minds in the same state, just enough to re-point the "most likely" pointer to the new substance instead of hydrogen, leading to huge savings overall. Or it could do something even more subversive, my imagination is weak.

That's what I was getting at, when I said a general problem with using sensory experiences as pointers is that the AI can influence sensory experiences.

Comment author: TheOtherDave 13 July 2012 04:55:12PM 2 points [-]

Well, right, but my point is that "the thing which differentially caused the sensory experiences to which I refer" does not refer to the same thing as "the thing which would differentially cause similar sensory experiences in the future, after you've made your changes," and it's possible to specify the former rather than the latter.

The AI can influence sensory experiences, but it can't retroactively influence sensory experiences. (Or, well, perhaps it can, but that's a whole new dimension of subversive. Similarly, I suppose a sufficiently powerful optimizer could rewrite the automaton rules in case #2, so perhaps we have a similar problem there as well.)

Comment author: cousin_it 13 July 2012 05:11:28PM *  1 point [-]

You need to describe the sensory experience as part of the AI's utility computation somehow. I thought it would be something like a bitstring representing a brain scan, which can refer to future experiences just as easily as past ones. Do you propose to include a timestamp? But the universe doesn't seem to have a global clock. Or do you propose to say something like "the values of such-and such terms in the utility computation must be unaffected by the AI's actions"? But we don't know how to define "unaffected" mathematically...

Comment author: TheOtherDave 13 July 2012 09:19:44PM *  1 point [-]

I was thinking in terms of referring to a brain. Or, rather, a set of them. But a sufficiently detailed brainscan would work just as well, I suppose.

And, sure, the universe doesn't have a clock, but a clock isn't needed, simply an ordering: the AI attends to evidence about sensory experiences that occurred before the AI received the instruction.

Of course, maybe it is incapable of figuring out whether a given sensory experience occurred before it received the instruction... it's just not smart enough. Or maybe the universe is weirder than I imagine, such that the order in which two events occur is not something the AI and I can actually agree on... which is the same case as "perhaps it can in fact retroactively influence sensory experiences" above.

Comment author: private_messaging 13 July 2012 03:13:45PM *  -3 points [-]

This means that equation (20) in Hutter is written as a utility function over sense data, where the reward channel is just a special case of sense data. We can easily adapt this equation to talk about any function computed directly over sense data - we can get AIXI to optimize any aspect of its sense data that we please. We can't get it to optimize a quality of the external universe. One of the challenges I listed in my FAI Open Problems talk, and one of the problems I intend to talk about in my FAI Open Problems sequence, is to take the first nontrivial steps toward adapting this formalism - to e.g. take an equivalent of AIXI in a really simple universe, with a really simple goal, something along the lines of a Life universe and a goal of making gliders, and specify something given unlimited computing power which would behave like it had that goal, without pre-fixing the ontology of the causal representation to that of the real universe, i.e., you want something that can range freely over ontologies in its predictive algorithms, but which still behaves like it's maximizing an outside thing like gliders instead of a sensory channel like the reward channel. This is an unsolved problem!

It gets more interesting if the computing power is not unlimited but strictly smaller than that of the universe in which the agent is living (excluding the ridiculous 'run sim since big bang and find yourself in it' non-solution). Also, it is not only an open problem in the FAI, but also an open problem in the dangerous uFAI.

edit: actually I would search for general impossibility proofs at that point. Also, keep in mind that having 'all possible models, weighted' is the ideal Bayesian approach, so it may be the case that simply striving for the most correct way of acting upon uncertainty makes it impossible to care about any real world goals.

Also it is rather interesting how 1 sample into 'unethical AI design space' (AIXI) yielded something which, most likely, is fundamentally incapable of caring about a real world goal (but is still an incredibly powerful optimization process if given enough computing power edit: i.e. AIXI doesn't care if you live or die but in a way quite different from a paperclip maximizer). In so much as one previously had an argument that such is incredibly unlikely, one ought to update and severely lower the probability of correctness of methods employed for generating that argument.

Comment author: Eliezer_Yudkowsky 13 July 2012 05:10:59PM 2 points [-]

The ontology problem has nothing to do with computing power, except that limited computing power means you use fewer ontologies. The number might still be large, and for a smart AI not fixable in advance; we didn't know about quantum fields just recently, and new approximations and models are being invented all the time. If your last paragraph isn't talking about evolution, I don't know what it's talking about.

Downvoting the whole thing as probable nonsense, though my judgment here is influenced by numerous downvoted troll comments that poster has made previously.

Comment author: private_messaging 13 July 2012 05:39:10PM *  0 points [-]

The ontology problem has nothing to do with computing power, except that limited computing power means you use fewer ontologies. The number might still be large, and for a smart AI not fixable in advance; we didn't know about quantum fields just recently, and new approximations and models are being invented all the time. If your last paragraph isn't talking about evolution, I don't know what it's talking about.

Limited computing power means that the ontologies have to be processed approximately (can't simulate everything at level of quarks all way from the big bang), likely in some sort of multi level model which can go down to level of quarks but also has to be able to go up to level of paperclips, i.e. would have to be able to establish relations between ontologies of different level of detail. It is not inconceivable that e.g. Newtonian mechanics would be part of any multi level ontology, no matter what it has at microscopic level. Note that while I am very skeptical about the AI risk, this is an argument slightly in favour of the risk.

Comment author: itaibn0 06 January 2014 11:43:20PM 1 point [-]

I believe AIXI is much more inspectable than you make it out to be. I think it is important to challenge your claim here because Holden appears to have trusted your expertise and hereby concede an important part of the argument.

AIXI's utility judgements are based a Solomonoff prior, which are based on the computer programs which return the input data. Computer programs are not black-boxes. A system implementing AIXI can easily also return a sample of typical expected future histories and the programs compressing these histories. By examining these programs, we can figure out what implicit model the AIXI system has of its world. These programs are optimized for shortness so they are likely to be very obfuscated, but I don't expect them to be incomprehensible (after all, they're not optimized for incomprehensibility). Even just sampling expected histories without their compressions is likely to be very informative. In the case of AIXItl the situation is better in the sense that it's output at any give time is guaranteed to be generated by just one length <l subprogram, and this subprogram comes with a proof justifying its utility judgement. It's also worse in that there is no way to sample its expected future histories. However, I expect the proof provided would implicitly contain such information. If either the programs or the proofs cannot be understood by humans, the programmers can just reject them and look at the next best candidates.

As for "What will be its effect on _?", this can be answered as well. I already stated that with AIXI you can sample future histories. This is because AIXI has a specific known prior it implements for its future histories, namely Solomonoff induction. This ability may seem limited because it only shows the future sensory data, but sensory data can be whatever you feed AIXI as input. If you want it to a have a realistic model of the world, this includes a lot of relevant information. For example, if you feed it the entire database of Wikipedia, it can give likely future versions of Wikipedia which already provides a lot of details on the effect of its actions.

Comment author: Squark 08 February 2013 06:51:51PM 0 points [-]

Regarding the question of formalizing an optimization agent with goals defined in terms of external universe rather than sensory input. It is possible to attack the problem by generalizing the framework I described in http://lesswrong.com/lw/gex/save_the_princess_a_tale_of_aixi_and_utility/8ekk for solving the duality problem. Specifically, consider an "initial guess" stochastic model of the universe including the machine on which our agent is running. I call it the "innate model" M. Now consider a stochastic process with the same degrees of freedom as M but governed by the Solomonoff semi-measure. This is the "unbiased model" S. The two can be combined by assigning transition probabilities proportional to the product of the probabilities assigned by M and S. If M is sufficiently "insecure" (in particular it doesn't assign 0 to any transition probability) then the resulting model S', considered as prior, allows arriving at any computable model after sufficient learning. Fix a utility function on the space of histories of our model (note that the histories include both intrinsic and extrinsic degrees of freedom). The intelligence I(A) of any given agent A (= program written in M in the initial state) can now be defined to be the expected utility of A in S'. We can now consider optimal or near-optimal agents in this sense (as opposed to the Legg-Hutter formalism for measuring intelligence, there is no guarantee there is a maximum rather than a supremum; unless of course we limit the length of the programs we consider). This is a generalization of the Legg-Hutter formalism which accounts for limited computational resources, solves the duality problem (such agents take into account possibly wireheading) and also provides a solution for the ontology problem. This is essentially a special case of the Orseau-Ring framework. It is however much more specific than Orseau-Ring where the prior is left completely unspecified. You can think of it as a recipe for constructing Orseau-Ring priors from realistic problems

Comment author: Squark 09 February 2013 01:28:52PM *  0 points [-]

I realized that although the idea of a deformed Solomonoff semi-measure is correct, the multiplication prescription I suggested is rather ad hoc. The following construction is a much more natural and justifiable way of combining M and S.

Fix t0 a time parameter. Consider a stochastic process S(-t0) that begins at time t = -t0, where t = 0 is the time our agent A "forms", governed by the Solomonoff semi-measure. Consider another stochastic process M(-t0) that begins from the initial conditions generated by S(-t0) (I'm assuming M only carries information about dynamics and not about initial conditions). Define S' to be the conditional probability distribution obtained from S by two conditions:

a. S and M coincide on the time interval [-t0, 0]

b. The universe contains A at time t=0

Thus t0 reflects the extent to which we are certain about M: it's like telling the agent we have been observing behavior M for time period t0.

There is an interesting side effect to this framework, namely that A can exert "acausal" influence on the utility by affecting the initial conditions of the universe (i.e. it selects universes in which A is likely to exist). This might seem like an artifact of the model but I think it might be a legitimate effect: if we believe in one-boxing in Newcomb's paradox, why shouldn't we accept such acausal effects?

For models with a concept of space and finite information velocity, like cellular automata, it might make sense to limit the domain of "observed M" in space as well as time, to A's past "light-cone"

Comment author: Eliezer_Yudkowsky 08 February 2013 06:59:48PM 0 points [-]

I cannot even slightly visualize what you mean by this. Please explain how it would be used to construct an AI that made glider-oids in a Life-like cellular automaton universe.

Comment author: Squark 08 February 2013 08:22:19PM 0 points [-]

Is the AI hardware separate from the cellular automaton or is it a part of it? Assuming the latter, we need to decide which degrees of freedom of the cellular automaton form the program of our AI. For example we can select a finite set of cells and allow setting their values arbitrarily. Then we need to specify our utility function. For example it can be a weighted sum of the number of gliders at different moments of time, or a maximum or whatever. However we need to make sure the expectation values converge. Then the "AI" is simply the assignment of values to the selected cells in the initial state which yields the maximal expect utility. Note though that if we're sure about the law governing the cellular automaton then there's no reason to use the Solomonoff semi-measure at all (except maybe as a prior for the initial state outside the selected cells). However if our idea of the way the cellular automaton works is only an "initial guess" then the expectation value is evaluated w.r.t. a stochastic process governed by a "deformed Solomonoff" semi-measure in which transitions illegal w.r.t. assumed cellular automaton law are suppressed by some factor 0 < p < 1 w.r.t. "pure" Solomonoff inference. Note that, contrary to the case of AIXI, I can only describe the measure of intelligence, I cannot constructively describe the agent maximizing this measure. This is unsurprising since building a real (bounded computing resources) AI is a very difficult problem

Comment author: MattMahoney 05 July 2012 05:38:30PM 0 points [-]

If we were smart enough to understand its policy, then it would not be smart enough to be dangerous.

Comment author: wedrifid 05 July 2012 06:13:31PM 1 point [-]

If we were smart enough to understand its policy, then it would not be smart enough to be dangerous.

That doesn't seem true. Simple policies can be dangerous and more powerful than I am.

Comment author: pleeppleep 12 June 2012 12:37:14PM *  26 points [-]

When I read posts like this I feel like an independent everyman watching a political debate.

The dialogue is oversimplified and even then I don't fully grasp exactly what's being said and the implications thereof, so I can almost feel my opinion shifting back and forth with each point that sounds sort of, kinda, sensible when I don't really have the capacity to judge the statements. I should probably try and fix that.

Comment author: Will_Newsome 12 June 2012 06:13:35PM 15 points [-]

The analogy is apt: blue-vs.-green politics aren't the only kind of politics, and debates over singularity policy have had big mind-killing effects on otherwise-pretty-rational LW folk before.

Comment author: John_Maxwell_IV 13 June 2012 01:45:52AM 1 point [-]

The core points don't strike me as being inherently difficult or technical, although Eliezer uses some technical examples.

Comment author: DanArmak 13 June 2012 08:33:10PM *  6 points [-]

The core points don't strike me as being inherently difficult or technical

That's precisely the problem, given that Eliezer is arguing that a technical appreciation of difficult problems is necessary to judge correctly on this issue. My understanding, like pleeppleep's, is limited to the simplified level given here, which means I'm reduced to giving weight to presentation and style and things being "kinda sensible".

Comment author: AlexMennen 13 June 2012 12:07:49AM *  9 points [-]

And if the preference function was just over the human's 'goodness' of the end result, rather than the accuracy of the human's understanding of the predictions, the AI might tell you something that was predictively false but whose implementation would lead you to what the AI defines as a 'good' outcome. And if we ask how happy the human is, the resulting decision procedure would exert optimization pressure to convince the human to take drugs, and so on.

I was under the impression that Holden's suggestion was more along the lines of: Make a model of the world. Remove the user from the model and replace it with a similar user that will always do what you recommend. Then manipulate this user so that it achieves its objective in the model, and report the actions that you have the user do in the model to the real user.

Thus, if the objective was to make the user happy, the Google Maps AGI would simply instruct the user to take drugs, rather than tricking him into doing so, because such instruction is the easiest way to manipulate the user in the model that the Google Maps AGI is optimizing in.

Comment author: Manfred 13 June 2012 01:20:18AM 8 points [-]

Actually, the easiest output for the AI in that case is "be happy."

Comment author: Eliezer_Yudkowsky 13 June 2012 01:28:21AM 2 points [-]

But - that's not what he meant!

Comment author: thomblake 13 June 2012 01:19:52PM 6 points [-]

I don't know why you keep harping on this. Just because an algorithm logically can produce a certain output, and probably will produce that output, doesn't mean good intentions and vigorous handwaving are any less capable of magic.

This is why when I fire a gun, I just point it in the general direction of my target, and assume the universe will know what I meant to hit.

Comment author: MBlume 21 June 2012 04:15:14AM 3 points [-]

I mean, it works in so many video games.

Comment author: Strange7 13 June 2012 08:14:13PM 3 points [-]

As a failure mode, "vague, useless, or trivially-obvious suggestions" is less of a problem than "rapidly eradicates all life." Historically, projects that were explicitly designed to be safe even when they inevitably failed have been more successful and less deadly than projects which were obsessively designed never to fail at all.

Comment author: AlexMennen 13 June 2012 05:26:49AM *  0 points [-]

Indeed it isn't, although I'm not so foolish as to claim to know how to fully specify my suggestion in a way that avoids all of these sorts of problems.

Comment author: Eliezer_Yudkowsky 13 June 2012 01:27:30AM 5 points [-]

Holden didn't actually suggest that. And while this suggestion is in a certain sense ingenious - it's not too far off from the sort of suggestions I flip through when considering how/if to implement CEV or similar processes - how do you "report the actions"? And do you report the reasons for them? And do you check to see if there are systematic discrepancies between consequences in the true model and consequences in the manipulated one? (This last point, btw, is sufficient that I would never try to literally implement this suggestion, but try to just structure preferences around some true model instead.)

Comment author: private_messaging 13 June 2012 04:39:29AM *  -1 points [-]

how do you "report the actions"?

How do you report the path the car should take? On the map. How do you report better transistor design? In the blueprint. How do we report software design? With UML diagram. (how do you report why that transistor works? Show simulator). It's just the most irreparable clinical psychopaths whom generate all outputs via extensive (and computationally expensive) modelling of the cognition (and decision process) of the listener. edit: i.e. modelling as to attain an outcome favourable to them; failing to empathise with listener that is failing to treat the listener as instance of self, but instead treating listener as a difficult to control servomechanism.

Comment author: Armok_GoB 13 June 2012 07:25:09PM *  1 point [-]

I can think of a bunch of random standard modes of display (top candidate: video and audio of what the simulated user sees and hears, plus subtitles of their internal model), and for the dispensaries you could run the simulation many times with random variations roughly along the same scope and dimensions as the differences between the simulations and reality, either just reacting plans that have to much divergence, or simply showing the display of all of them (wich'd also help against frivolous use if you have to watch the action 1000 times before doing it). I'd also say make the simulated user a total drone with seriously rewired neurology to try to always and only do what the AI tells it to.

Not that this solves the problem - I've countered the real dangerous things I notice instantly, but 5 mins to think of it and I'll notice 20 more - but I though someone should actually try to answer the question in spirit and letter and most charitable interpretation.

also, it'd make a nice movie.

Comment author: Strange7 13 June 2012 02:29:20AM 1 point [-]

Hardwiring the AI to be extremely naive about how easy the user is to manipulate might not be sufficient for safety, but it does seem like a pretty good start.

Comment author: Eliezer_Yudkowsky 13 June 2012 03:02:18AM 10 points [-]

Delete the word "hardwiring" from your vocabulary. You can't do it with wires, and saying it doesn't accomplish any magic.

Comment author: cousin_it 13 June 2012 01:41:39PM *  6 points [-]

I think there is an interpretation of "hardwiring" that makes sense when talking about AI. For example, say you have a chess program. You can make a patch for it that says "if my light squared bishop is threatened, getting it out of danger is highest priority, second only to getting the king out of check". Moreover, even for very complex chess programs, I would expect that patch to be pretty simple, compared to the whole program.

Maybe a general AI will necessarily have an architecture that makes such patches impossible or ineffective. Then again, maybe not. You could argue that an AI would work around any limitations imposed by patches, but I don't see why a computer program with an ugly patch would magically acquire a desire to behave as if it didn't have the patch, and converge to maximizing expected utility or something. In any case I'd like to see a more precise version of that argument.

ETA: I share your concern about the use of "hardwiring" to sweep complexity under the rug. But saying that AIs can do one magical thing (understand human desires) but not another magical thing (whatever is supposed to be "hardwired") seems a little weird to me.

Comment author: Eliezer_Yudkowsky 13 June 2012 06:19:29PM 9 points [-]

Yeah, well, hardwiring the AI to understand human desires wouldn't be goddamned trivial either, I just decided not to go down that particular road, mostly because I'd said it before and Holden had apparently read at least some of it.

Getting the light-square bishop out of danger as highest priority...

1) Do I assume the opponent assigns symmetric value to attacking the light-square bishop?

2) Or that the opponent actually values checkmates only, but knows that I value the light-square bishop myself and plan forks and skewers accordingly?

3) Or that the opponent has no idea why I'm doing what I'm doing?

4) Or that the opponent will figure it out eventually, but maybe not in the first game?

5) What about the complicated static-position evaluator? Do I have to retrain all of it, and possibly design new custom heuristics, now that the value of a position isn't "leads to checkmate" but rather "leads to checkmate + 25% leads to bishop being captured"?

Adding this to Deep Blue is not remotely as trivial as it sounds in English. Even to add it in a half-assed way, you have to at least answer question 1, because the entire non-brute-force search-tree pruning mechanism depends on guessing which branches the opponent will prune. Look up alpha-beta search to start seeing why everything becomes more interesting when position-values are no longer being determined symmetrically.

Comment author: cousin_it 13 June 2012 07:11:58PM *  10 points [-]

For what it's worth, the intended answers are 1) no 2) no 3) yes 4) no 5) the evaluation function and the opening book stay the same, there's just a bit of logic squished above them that kicks in only when the bishop is threatened, not on any move before that.

Yeah, game-theoretic considerations make the problem funny, but the intent wasn't to convert an almost-consistent utility maximizer into another almost-consistent utility maximizer with a different utility function that somehow values keeping the bishop safe. The intent was to add a hack that throws consistency to the wind, and observe that the AI doesn't rebel against the hack. After all, there's no law saying you must build only consistent AIs.

My guess is that's what most folks probably mean when they talk about "hardwiring" stuff into the AI. They don't mean changing the AI's utility function over the real world, they mean changing the AI's code so it's no longer best described as maximizing such a function. That might make the AI stupid in some respects and manipulable by humans, which may or may not be a bad thing :-) Of course your actual goals (whatever they are) would be better served by a genuine expected utility maximizer, but building that could be harder and more dangerous. Or at least that's how the reasoning is supposed to go, I think.

Comment author: Wei_Dai 14 June 2012 02:55:52AM 5 points [-]

The intent was to add a hack that throws consistency to the wind, and observe that the AI doesn't rebel against the hack.

Why doesn't the AI reason "if I remove this hack, I'll be more likely to win?" Because this is just a narrow chess AI and the programmer never gave it general reasoning abilities?

Comment author: private_messaging 26 June 2012 10:45:59AM *  1 point [-]

Why doesn't the AI reason "if I remove this hack, I'll be more likely to win?"

More interesting question is why it (if made capable of such reflection) would not take it a little step further and ponder what happens if it removes enemy's queen from it's internal board, which would also make it more likely to win, with its internal definition of win which is defined in terms of internal board.

Or why would anyone go through the bother of implementing possibly irreducible notion of what 'win' really means in the real world, given that this would simultaneously waste computing power on unnecessary explorations and make AI dangerous / uncontrollable.

Thing is, you don't need to imagine the world dying to avoid making pointless likely impossible accomplishments.

Comment author: cousin_it 14 June 2012 07:39:24AM *  0 points [-]

Yeah, because it's just a narrow real-world AI without philosophical tendencies... I'm actually not sure. A more precise argument would help, something like "all sufficiently powerful AIs will try to become or create consistent maximizers of expected utility, for such-and-such reasons".

Comment author: Vladimir_Nesov 14 June 2012 08:19:14AM *  4 points [-]

Does a pair of consistent optimizers with different goals have a tendency to become a consistent optimizer?

The problem with powerful non-optimizers seems to be that the "powerful" property already presupposes optimization power, and so at least one optimizer-like thing is present in the system. If it's powerful enough and is not contained, it's going to eat all the other tendencies of its environment, and so optimization for its goal will be all that remains. Unless there is another optimizer able to defend its non-conformity from the optimizer in question, in which case the two of them might constitute what counts as not-a-consistent-optimizer, maybe?

Comment author: Eliezer_Yudkowsky 13 June 2012 08:18:01PM 0 points [-]

Option 3? Doesn't work very well. You're assuming the opponent doesn't want to threaten the bishop, which means you yank it to a place where it would be safe if the opponent doesn't want to threaten it, but if the opponent clues in, it's then trivial for them to threaten the bishop again (to gain more advantage as you try to defend), which you weren't expecting them to do, because that's not how your search tree was structured. Kasparov would kick hell out of thus-hardwired Deep Blue as soon as he realized what was happening.

It's that whole "see the consequences of the math" thing...

Comment author: cousin_it 14 June 2012 07:46:51AM 14 points [-]

Either your comment is in violent agreement agreement with mine ("that might make the AI stupid in some respects and manipulable by humans"), or I don't understand what you're trying to say...

Comment author: Eliezer_Yudkowsky 14 June 2012 09:23:08PM 4 points [-]

Probably violent agreement.

Comment author: Strange7 13 June 2012 05:10:54AM 6 points [-]

I was sorely tempted, upon being ordered to self-modify in such a way, to respond angrily. It implies a lack of respect for the integrity of those with whom you are trying to communicate. You could have said "taboo" instead of demanding a permanent loss.

Do you think it would be outright impossible, to handicap an AI in such a way that it cannot conceive of a user interpreting it's advice in any but the most straightforward way, and therefore eschews manipulative output? Do you think it would be useless as a safety feature? Do you think it would be unwise for some other reason, some unintended consequence? Or are you simply objecting to my phrasing?

Comment author: Johnicholas 13 June 2012 12:15:12PM 4 points [-]

The distinction between hardwiring and softwiring is, at above the most physical, electronic aspects of computer design, a matter of policy - something in the programmer's mind and habits, not something out in the world that the programmer is manipulating. From any particular version of the software's perspective, all of the program it is running is equally hard (or equally soft).

It may not be impossible to handicap an entity in some way analogous to your suggestion, but holding fiercely to the concept of hardwiring will not help you find it. Thinking about mechanisms that would accomplish the handicapping in an environment where everything is equally hardwired would be preferable.

There's some evidence that chess AIs 'personality' (an emergent quality of their play) is related to a parameter of their evaluation function called 'contempt', which is something like (handwaving wildly) how easy the opponent is to manipulate. In general, AIs with higher contempt seek to win-or-lose more, and seek to draw less. What I'm trying to say is, your idea is not without merit, but it may have unanticipated consequences.

Comment author: Eliezer_Yudkowsky 13 June 2012 06:26:01PM 6 points [-]

I'm saying that using the word "hardwiring" is always harmful because they imagine an instruction with lots of extra force, when in fact there's no such thing as a line of programming which you say much more forcefully than any other line. Either you know how to program something or you don't, and it's usually much more complex than it sounds even if you say "hardwire". See the reply above on "hardwiring" Deep Blue to protect the light-square bishop. Though usually it's even worse than this, like trying to do the equivalent of having an instruction that says "#define BUGS OFF" and then saying, "And just to make sure it works, let's hardwire it in!"

Comment author: Strange7 13 June 2012 08:04:24PM 3 points [-]

There is, in fact, such a thing as making some parts of the code more difficult to modify than other parts of the code.

I apologize for having conveyed the impression that I thought designing an AI to be specifically, incurably naive about how a human querent will respond to suggestions would be easy. I have no such misconception; I know it would be difficult, and I know that I don't know enough about the relevant fields to even give a meaningful order-of-magnitude guess as to how difficult. All I was suggesting was that it would be easier than many of the other AI-safety-related programming tasks being discussed, and that the cost-benefit ratio would be favorable.

Comment author: Eliezer_Yudkowsky 13 June 2012 08:19:44PM 1 point [-]

There is, in fact, such a thing as making some parts of the code more difficult to modify than other parts of the code.

There is? How?

Comment author: Strange7 13 June 2012 08:29:08PM *  3 points [-]
Comment author: Johnicholas 14 June 2012 02:56:48AM 2 points [-]

I don't think Strange7 is arguing Strange7's point strongly; let me attempt to strengthen it.

A button that does something dangerous, such as exploding bolts that separate one thing from another thing, might be protected from casual, accidental changes by covering it with a lid, so that when someone actually wants to explode those bolts, they first open the lid and then press the button. This increases reliability if there is some chance that any given hand motion is an error, but the errors of separate hand motions are independent. Similarly 'are you sure' dialog boxes.

In general, if you have several components, each of a given reliability, and their failure modes are somewhat independent, then you can craft a composite component of greater reliability than the individuals. The rings that Strange7 brings up are an example of this general pattern (there may be other reasons why layers-of-rings architectures are chosen for reliability in practice - this explanation doesn't explain why the rings are ordered rather than just voting or something - this is just one possible explanation).

Comment author: Eliezer_Yudkowsky 14 June 2012 03:13:54AM 3 points [-]

This is reasonable, but note that to strengthen the validity, the conclusion has been weakened (unsurprisingly). To take a system that you think is fundamentally, structurally safe and then further build in error-delaying, error-resisting, and error-reporting factors just in case - this is wise and sane. Calling "adding impediments to some errors under some circumstances" hardwiring and relying on it as a primary guarantee of safety, because you think some coded behavior is firmly in place locally independently of the rest of the system... will usually fail to cash out as an implementable algorithm, never mind it being wise.

Comment author: Strange7 14 June 2012 03:23:36AM 4 points [-]

The conclusion has to be weakened back down to what I actually said: that it might not be sufficient for safety, but that it would probably be a good start.

Comment author: pnrjulius 19 June 2012 04:08:15AM 0 points [-]

Don't programmers do this all the time? At least with current architectures, most computer systems have safeguards against unauthorized access to the system kernel as opposed to the user documents folders...

Isn't that basically saying "this line of code is harder to modify than that one"?

In fact, couldn't we use exactly this idea---user access protocols---to (partially) secure an AI? We could include certain kernel processes on the AI that would require a passcode to access. (I guess you have to stop the AI from hacking its own passcodes... but this isn't a problem on current computers, so it seems like we could prevent it from being a problem on AIs as well.)

Comment author: RichardWein 14 July 2012 06:37:08PM 0 points [-]

[Responding to an old comment, I know, but I've only just found this discussion.]

Never mind special access protocols, you could make code unmodifiable (in a direct sense) by putting it in ROM. Of course, it could still be modified indirectly, by the AI persuading a human to change the ROM. Even setting aside that possibility, there's a more fundamental problem. You cannot guarantee that the code will have the expected effect when executed in the unpredictable context of an AGI. You cannot even guarantee that the code in question will be executed. Making the code unmodifiable won't achieve the desired effect if the AI bypasses it.

In any case, I think the whole discussion of an AI modifying its own code is rendered moot by the fuzziness of the distinction between code and data. Does the human brain have any code? Or are the contents just data? I think that question is too fuzzy to have a correct answer. An AGI's behaviour is likely to be greatly influenced by structures that develop over time, whether we call these code or data. And old structures need not necessarily be used.

AGIs are likely to be unpredictable in ways that are very difficult to control. Holden Karnofsky's attempted solution seems naive to me. There's no guarantee that programming an AGI his way will prevent agent-like behaviour. Human beings don't need an explicit utility function to be agents, and neither does an AGI. That said, if AGI designers do their best to avoid agent-like behaviour, it may reduce the risks.

Comment author: bogdanb 10 July 2013 10:03:04PM *  0 points [-]

I always thought that "hardwiring" meant implementing [whatever functionality is discussed] by permanently (physically) modifying the machine, i.e. either something that you couldn’t have done with software, or something that prevents the software from actually working in some way it did before. The concept is of immutability within the constraints, not of priority or "force".

Which does sound like something one could do when they can’t figure out how to do the software right. (Watchdogs are pretty much exactly that, though some or probably most are in fact programmable.)

Note that I’m not arguing that the word is not harmful. It just seemed you have a different interpretation of what that word suggests. If other people use my interpretation (no idea), you might be better at persuading it if you address that.

I’m quite aware that from the point of view of a godlike AI, there’s not much difference between circumventing restrictions in its software and (some kinds of) restrictions in hardware. After all, the point of FAI is to get it to control the universe around it, albeit to our benefit. But we’re used to computers not having much control over their hardware. Hell, I just called it “godlike” and my brain still insists to visualize it as a bunch of boxes gathering dust and blinking their leds in a basement.

And I can’t shake the feeling that between "just built" and "godlike" there’s supposed to be quite a long time when such crude solutions might work. (I’ve seen a couple of hard take-off scenarios, but not yet a plausible one that didn’t need at least a few days of preparation after becoming superhuman.)

Imagine we took you, gave you the best "upgrades" we can do today plus a little bit (say, a careful group of experts figuring out your ideal diet of nootropics, training you to excellence everything from acting to martial arts, and gave you nanotube bones and a direct internet link to your head). Now imagine you have a small bomb in your body, set to detonate if tampered with or if one of several remotes distributed throughout the population is triggered. The worlds best experts tried really hard to make it fail-deadly.

Now, I’m not saying you couldn’t take over the world, send all men to Mars and the women to Venus, then build a volcano lair filled with kittens. But it seems far from certain, and I’m positive it’d take you a long time to succeed. And, it does feel that a new-born AI would like that for a while rather than turn into Prime Intellect in five minutes. (Again, this is not an argument that UFAI is no problem. I guess I’m just figuring out why it seems that way to mostly everyone.)

[Huh, I just noticed I’m a year late on this chat. Sorry.]

Comment author: Eliezer_Yudkowsky 11 July 2013 12:08:35AM 4 points [-]

Software physically modifies the machine. What can you do with a soldering iron that you can't do with a program instruction, particularly with respect to building a machine agent? Either you understand how to write a function or you don't.

Comment author: bogdanb 11 July 2013 08:00:50PM *  1 point [-]

That is all true in principle, but in practice it’s very common that one of the two is not feasible. For example, you can have a computer. You can program the computer to tell you when it’s reading from the hard drive, or communicates to the network, say by blinking an LED. If the program has a bug (e.g., it’s not the kind of AI you wanted to build), you might not be notified. But you can use a soldering iron to electrically link the LED to the relevant wires, and it seems to most users that no possible programming bug can make the LED not light up when it should.

Of course, that’s like the difference between programming a robot to stay in a pen, or locking the gate. It looks like whatever bug you could introduce in the robot’s software cannot cause the robot to leave. Which ignores the fact that robot might learn to climb the fence, make a key, convince someone else (or hack an outside robot) to unlock the gate.

I think most people would detect the dangers in the robot case (because they can imagine themselves finding a way to escape), but be confused by the AI-in-the-box one (simply because it’s harder to imagine yourself as software, and even if you manage to you’d still have much fewer ideas come to mind, simply because you’re not used to being software).

Hell, most people probably won’t even have the reflex to imagine themselves in place of the AI. My brain reflexively tells me "I can’t write a program to control that LED, so even if there’s a bug it won’t happen". If instead I force myself to think "How would I do that if I were the AI", it’s easier to find potential solutions, and it also makes it more obvious that someone else might find one. But that may be because I’m a programmer, I’m not sure if it applies to others.

Comment author: Pentashagon 14 June 2012 07:03:39PM 0 points [-]

Presumably a well-designed agent will have nearly infallible trust in certain portions of its code and data, for instance a theorem prover/verifier and the set of fundamental axioms of logic it uses. Manual modifications at that level would be the most difficult for an agent to change, and changes to that would be the closest to the common definition of "hardwiring". Even a fully self-reflective agent will (hopefully) be very cautious about changing its most basic assumptions. Consider the independence of the axiom of choice from ZF set theory. An agent may initially accept choice or not but changing whether it accepts it later is likely to be predicated on very careful analysis. Likewise an additional independent axiom "in games of chess always protect the white-square bishop" would probably be much harder to optimize out than a goal.

Or from another angle wherever friendliness is embodied in a FAI would be the place to "hardwire" a desire to protect the white-square bishop as an additional aspect of friendliness. That won't work if friendliness is derived from a concept like "only be friendly to cognitive processes bearing a suitable similarity to this agent" where suitable similarity does not extend to inanimate objects, but if friendliness must encode measurable properties of other beings then it might be possible to sneak white-square bishops into that class, at least for a (much) longer period than artificial subgoals would last.

Comment author: Stuart_Armstrong 13 June 2012 01:57:22PM 0 points [-]

Feels like "utility indifference" could be used to get something like that.

Comment author: [deleted] 30 June 2012 05:26:23AM 0 points [-]

What is the mathematical implementation of indifference?

Armstrong suggests that you implement it as a special value that is exactly equal to every other utility value. So if the AI comes up with an outcome that would yield +751 utility, it would treat being destroyed as having +751 utility. Whatever algorithm you choose for resolving ties determines whether the AI suicides (by doing something that causes its human managers to kill it).

Let's see how this works with a hypothetical example. Our good old friend the Paperclip Maximizer to the rescue!

Papey wants more paperclips. It compares several possibilities. In one, it generates 1,000 paperclips before its human operators kill it. In another, it generates 999 paperclips, but its human operators leave it alive, and it will have future chances to create paperclips. It expects to remain alive long enough to generate exactly two additional paperclips this way.

Now, Papey's decision algorithm chooses between outcomes of equal utility in a uniformly random manner. Papey has two possible outcomes right now: suicide (for 1001 pseudo-paperclips, since suicide is always worth as much as any other decision) or generate 999 paperclips now and an expected 2 paperclips later (for 1001 paperclips).

At this point, Papey will, with 50% probability, generate 999 paperclips; otherwise, Papey will do something to cause his operators to kill him. (Not sure what, but presumably it won't involve destroying paperclips.) Poor Papey. There's a 99% probability that he'll suicide within seven decisions.

Now, if the tie breaking algorithm isn't random, in a given situation, it will either go with suicide deterministically or go against suicide deterministically. You can always go against suicide, in which case you may as well omit utility indifference entirely. Or you can always go for suicide, in which case you found a fancy way of wasting money.

Or you could do something interesting. But if you had something to put here that guaranteed a good result, you could use that as the utility function.

Friendly AI is indifferent to utility indifference.

Comment author: Stuart_Armstrong 01 July 2012 08:44:06AM *  0 points [-]

I wasn't advocating using utility indifference to death for this example, more "utility indifference as to whether the user ever reads its output" or something along those lines.

But let's look at your example more in detail. You say the AI has two choices: it makes 1000 paperclips and is killed, or it makes 999 and is left alive, and then goes on to make another 2.

In this scenario, there is only one case of death, coming from the first decision. So utility indifference does not change anything about the impact of the second decision. What indifference means is that if humans try and blow Papey up after the 1000 paper clips, it will derive as much utility from a successful blow up as from a failed blow up (in less correct but more intuitive terms, you could say "Papey acts as if it doesn't believe the explosives will work").

So how many paper clips could Papey make if the explosives fail? Let's assume another 2, for a grand total of 1000.

So the paperclip-utility of the first course is is 1002, that of the second is 1001: it will thus take the first course. Hence it will misbehave visibly (rather than keeping that a secret), and will likely end up blown up by us.

Now imagine it could get rid of the explosives at no costs. Here, your point has merit: Papey is indifferent between removing the explosives or not removing it, and it tie breaker algorithm will determine what it does. But let's make this more realistic: removing the explosives has a tiny cost, say a 50% chance it will fail to manufacture one paperclip. So now it can:

1) Misbehave, leave explosives in place: expected utility 1002.

2) Pretend to behave: expected utility 999+2 = 1001.

3) Misbehave, remove explosives in place: expected utility 1002-0.5=1001.5.

Hence Papey will choose the first option, we get to see that it misbehaves, and we likely blow it up. That was the initial point of utility indifference.

But as I said here, we wouldn't be using utility indifference in this way. It would more be to make the AI "behave as if it believed that nobody would get to read its printout."

Comment author: [deleted] 04 July 2012 09:59:12PM 0 points [-]

it will derive as much utility from a successful blow up as from a failed blow up (in less correct but more intuitive terms, you could say "Papey acts as if it doesn't believe the explosives will work").

Those are quite different. In one case, you're adjusting its utility function. In the other, you're adjusting its expectations. (And the remainder of your post indicates that you intend to adjust the AI's expectations of what will happen, and don't care about its utility function.)

Now, that might work. If the AI is self-modifying, then it might notice that that part of its planning function is obviously incorrect and fix it. It might simulate human responses to its proposed actions and notice that there are problematic conflicts around that area. But there's at least a chance that the AI will not notice that.

I wouldn't want to trust my life to it, but it's got a shot of working.

Comment author: Stuart_Armstrong 05 July 2012 10:44:27AM *  0 points [-]

The method is entirely about adjusting the utility function, not the expectations, as I explained in the paper. And the adjustments are ideally to be made from the very start, so that the AI never has cause to fear subsequent adjustment to its utility.

Comment author: [deleted] 08 July 2012 07:39:54AM *  0 points [-]

Rereading the paper, it looks like you're mucking about with utility, not expected utility. So let's see how this works out in practice.

You assign the AI a task of finding a cure for cancer. This will net it 10 utilons if it succeeds. Thanks to your utility indifference module, it will also net 10 utilons if it does not find a cure for cancer and you trigger the explosives to kill the AI.

The AI does a preliminary estimate to see its odds of success for the relevant outcomes. It can find a cure for cancer with 80% likelihood. It can cause you to trigger the explosives and kill the AI with 90% likelihood.

The expected utility for attempting to cure cancer is 8. The expected utility for attempting suicide is 9.

It seems pretty straightforward and easy to pose as a mad, dangerous AI about to destroy the world, at least well enough to make a human panic and hit the emergency destruct button. I'm sure I could convince you to kill me, if you just had to push a button. It seems a lot less certain that the AI would be able to solve the problems given it, since those problems will be things that humans haven't been able to solve, and since the AI will be under various other restrictions to keep it safe.

Comment author: Stuart_Armstrong 08 July 2012 04:05:08PM 0 points [-]

You assign the AI a task of finding a cure for cancer. This will net it 10 utilons if it succeeds. Thanks to your utility indifference module, it will also net 10 utilons if it does not find a cure for cancer and you trigger the explosives to kill the AI.

The AI gets nothing from you deciding to kill it. What utility indifference means is that it will get the same utility if you decide to kill it, and the explosives go off, as if you decided to kill it, and the explosives failed to go off.

So if it expects to cure cancer with 80% likelihood if it survives, and thus get 8 utilitons, then if if dies from the explosion, it will also get 8 utilitons.

More precisely there are six worlds, depending on a) whether we trigger the explosives or not, b) whether it survives the explosion or not, and c) whether cancer is curable by an unmolested AI. In obvious notation, these are TSC, TS¬C, T¬SC, T¬S¬C, ¬TC, ¬T¬C. We are assuming in your example that U(¬T¬C)=U(TS¬C)=0 and U(¬TC)=U(TSC)=10. Utility indifference then sets U(T¬S¬C)=0 and U(T¬SC)=10.

Hence assuming that S is independent of C (the curability of cancer by an unmolested AI is independent of whether particular explosives go off), we see that the AI is indifferent as to whether the explosives go off - it gains nothing either way, so has no cause to influence the explosives in any direction. If we further assume T and C are independent (the curability of cancer by an unmolested AI is independent of whether we decide to blow up this particular AI), then the AI also has no preference as to whether we trigger the explosives or not - it gains nothing either way.

Comment author: [deleted] 08 July 2012 07:52:38PM 1 point [-]

Ah, that makes sense. It isn't indifferent to suicide as such; it's only indifferent to your success at attempting to kill it, should you make the attempt.

Thanks for your patience!

Comment author: Stuart_Armstrong 09 July 2012 10:59:51AM 0 points [-]

No prob :-) Always happy when I manage to explain something successfully!

Comment author: Nick_Beckstead 12 June 2012 02:14:57PM 8 points [-]

There are two ways to read Holden's claim about what happens if 100 experts check the proposed FAI safety proof. On one reading, Holden is saying that if 100 experts check it and say, "Yes, I am highly confident that this is in fact safe," then activating the AI kills us all with 90% probability. On the other reading, Holden is saying that even if 100 experts do their best to find errors and say, "No, I couldn't identify any way in which this will kill us, though that doesn't mean it won't kill us," then activating the AI kills us all with 90% probability. I think the first reading is very implausible. I don't believe the second reading, but I don't think it's obviously wrong. I think the second reading is the more charitable and relevant one.

Comment author: Nick_Beckstead 12 June 2012 05:32:20PM 1 point [-]

For context, I pointed this out because it looks like Eliezer is going for the first reading and criticizing that.

Comment author: Eliezer_Yudkowsky 12 June 2012 06:04:15PM 7 points [-]

Nope, I was assuming the second reading. The first reading is too implausible to be considered at all.

Comment author: Nick_Beckstead 12 June 2012 06:13:22PM 5 points [-]

Good. But now I find this response less compelling:

If Holden says there's 90% doom probability left over no matter what sane intelligent people do (all of which goes away if you just build Google Maps AGI, but leave that aside for now) I would ask him what he knows now, in advance, that all those sane intelligent people will miss. I don't see how you could (well-justifiedly) access that epistemic state. [emphasis added]

Holden might think that these folks will be of the opinion, "I can't see an error, but I'm really not confident that there isn't an error." He doesn't have to think that he knows something they don't. In particular, he doesn't have to think that there is some special failure mode he's thought of that none of them have thought of.

Comment author: Eliezer_Yudkowsky 12 June 2012 08:58:37PM 4 points [-]

Nonetheless, where is he getting the 90% doom probability from?

Comment author: Nick_Beckstead 12 June 2012 09:21:34PM *  1 point [-]

I'm with you, 90% seems too high given the evidence he cites or any evidence I know of.

Comment author: Arepo 13 June 2012 12:01:52PM 4 points [-]

Assuming you accept the reasoning, 90% seems quite generous to me. What percentage of complex computer programmes when run for the first time exhibit behaviour the programmers hadn't anticipated? I don't have much of an idea, but my guess would be close to 100. If so, the question is how likely unexpected behaviour is to be fatal. For any programme that will eventually gain access to the world at large and quickly become AI++, that seems (again, no data to back this up - just an intuitive guess) pretty likely, perhaps almost certain.

For any parameter of human comfort (eg 253 degrees Kelvin, 60% water, 40 hour working weeks), a misplaced decimal point misplaced by seems like it would destroy the economy at best and life on earth at worst.

If Holden’s criticism is appropriate, the best response might be to look for other options rather than making a doomed effort to make FAI – for example trying to prevent the development of AI anywhere on earth, at least until we can self-improve enough to keep up with it. That might have a low probability of success, but if FAI has sufficiently low probability, it would still seem like a better bet.

Comment author: TheOtherDave 13 June 2012 01:30:06PM 4 points [-]

You know, the idea that SI might at any moment devote itself to suppressing AI research is one that pops up from time to time, the logic pretty much being what you suggest here, and until this moment I have always treated it as a kind of tongue-in-cheek dig at SI.

I have only just now come to realize that the number of people (who are not themselves affiliated with SI) who really do seem to consider suppressing AI research to be a reasonable course of action given the ideas discussed on this forum has a much broader implication in terms of the social consequences of these ideas. That is, I've only just now come to realize that what the community of readers does is just as important, if not more so, than what SI does.

I am now becoming genuinely concerned that, by participating in a forum that encourages people to take seriously ideas that might lead them to actively suppress AI research, I might be doing more harm than good.

I'll have to think about that a bit more.

Arepo, this is not particularly directed at you; you just happen to be the data point that caused this realization to cross an activation threshold.

Comment author: Bruno_Coelho 15 June 2012 01:09:38AM 0 points [-]

People with similar background are entering in AI field because they like reduce x-risks, so it's not obvious this is happening. If safety guided research supress AI research, then be it. Extremely rapid advance per se is not good, if the consequence is extiction.

Comment author: falenas108 13 June 2012 04:16:57PM 3 points [-]

What percentage of complex computer programmes when run for the first time exhibit behaviour the programmers hadn't anticipated? I don't have much of an idea, but my guess would be close to 100.

That's for normal programs, where errors don't matter. If you look at ones where people carefully look over the code because lives are at stake (like NASA rockets), then you'll have a better estimate.

Probably still not accurate, because much more is at stake for AI than just a few lives, but it will be closer.

Comment author: TheOtherDave 13 June 2012 04:28:25PM 2 points [-]

I suspect that unpacking "run a program for the first time" more precisely would be useful here; it's not clear to me that everyone involved in the conversation has the same referents for it.

Comment author: Nick_Beckstead 13 June 2012 06:59:23PM *  1 point [-]

This. I see that if you have one and only one chance to push the Big Red Button and you're not allowed to use any preliminary testing of components or boxing strategies (or you're confident that those will never work) and you don't get most of the experts to agree that it is safe, then 90% is more plausible. If you envision more of these extras to make it safer--which seems like the relevant thing to envision--90% seems too high to me.

Comment author: DanArmak 13 June 2012 08:29:43PM 1 point [-]

Surely NASA code is thoroughly tested in simulation runs. It's the equivalent of having a known-perfect method of boxing an AI.

Comment author: asparisi 14 June 2012 11:18:31PM 0 points [-]

Huh. This brings up the question of whether or not it would be possible to simulate the AGI code in a test-run without regular risks. Maybe create some failsafe that is invisible to the AGI that destroys it if it is "let out of the box" or (to incorporate Holden's suggestion, since it just came to me) having a "tool mode" where the AGI's agent-properties (decision making, goal setting, etc.) are non-functional.

Comment author: Randaly 13 June 2012 09:12:48PM 0 points [-]

The last three versions of [NASA's] program -- each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors.

From They Write the Right Stuff

Note, however, that a) this is after many years of debugging from practice, b) NASA was able to safely 'box' their software, and c) even one error, if in the wrong place, would be really bad.

Comment author: Strange7 13 June 2012 09:30:37PM 0 points [-]

How hard would it actually be to "box" an AI that's effectively had it's brain sliced up into very small chunks?

A program could, if it was important enough and people were willing to take the time to do so, be broken down into pieces and each of the pieces tested separately. Any given module has particular sorts of input it's designed to receive, and particular sorts of output it's supposed to pass on to the next module. Testers give the module different combinations of valid inputs and try to get it to produce an invalid output, and when they succeed, either the module is revised and the testing process on that module starts over from the beginning, or the definition of valid inputs is narrowed, which changes the limits for valid outputs and forces some other module further back to be redesigned and retested. A higher-level analysis, which is strictly theoretical, also tries to come up with sequences of valid inputs and outputs which could lead to a bad outcome. Eventually, after years of work and countless iterations of throwing out massive bodies of work to start over, you get a system which is very tightly specified to be safe, and meets those specs under all conceivable conditions, but has never actually been plugged in and run as a whole.

Comment author: JenniferRM 12 June 2012 08:19:10PM *  4 points [-]

It seems like this is turning into a statement about human technical politics.

Yes, I am highly confident that this is in fact safe.

No, I couldn't identify any way in which this will kill us, though that doesn't mean it won't kill us.

The latter is stereotypically something a cautious engineer in cover-your-ass-mode is likely to say no matter how much quality assurance has happened. The first is something that an executive in selling-to-investors-and-the-press-mode is likely to say once they estimate it will have better outcomes than saying something else with the investors and the press, perhaps just because they know of something worse that will happen outside their control that seems very likely to be irreversible and less likely to be good. Between these two stereotypes lays a sort of "reasonable rationalist speaking honestly but pragmatically"?

This is a hard area to speak about clearly between individuals without significant interpersonal calibration on the functional meaning of "expert", because you run into Dunning-Kruger effects if you aren't careful and a double illusion of transparency can prevent you from even noticing the miscommunication.

There are conversations that can allow specific people to negotiate a common definition with illustrations grounded in personal experience here, but they take many minutes or hours, and are basically a person-to-person protocol. The issue is doubly hard with a general audience because wildly different gut reactions will be elicited and there will be bad faith participation by at least some people, and so on. Rocket scientists get this wrong sometimes. It is a hard problem.

Comment author: lukeprog 12 June 2012 04:20:18AM 18 points [-]

To clarify, for everyone:

There are now three "major" responses from SI to Holden's Thoughts on the Singularity Institute (SI): (1) a comments thread on recent improvements to SI as an organization, (2) a post series on how SI is turning donor dollars into AI risk reduction and how it could do more of this if it had more funding, and (3) Eliezer's post on Tool AI above.

At least two more major responses from SI are forthcoming: a detailed reply to Holden's earlier posts and comments on expected value estimates (e.g. this one), and a long reply from me that summarizes my responses to all (or almost all) of the many issues raised in Thoughts on the Singularity Institute (SI).

Comment author: Nick_Beckstead 12 June 2012 02:18:10PM 2 points [-]

How much of this is counting toward the 50,000 words of authorized responses?

Comment author: lukeprog 12 June 2012 03:18:25PM 2 points [-]

I told Holden privately that this would be explained in my final "summary" reply. I suspect the 5200 words of Eliezer's post above will be part of the 50,000.

Comment author: Wei_Dai 29 August 2012 08:55:38PM 1 point [-]

Luke, do you know if there has been any official (or unofficial) response to my argument that Holden quoted in his post?

Comment author: lukeprog 29 August 2012 09:36:16PM *  1 point [-]

Not that I know of. I fully agree with that comment, and I suspect Eliezer does as well.

Comment author: Kaj_Sotala 12 June 2012 09:10:50PM *  16 points [-]

Software that does happen to interface with humans is selectively visible and salient to humans, especially the tiny part of the software that does the interfacing; but this is a special case of a general cost/benefit tradeoff which, more often than not, turns out to swing the other way, because human advice is either too costly or doesn't provide enough benefit.

I suspect this is the biggest counter-argument for Tool AI, even bigger than all the technical concerns Eliezer made in the post. Even if we could build a safe Tool AI, somebody would soon build an agent AI anyway.

My five cents on the subject, from something that I'm currently writing:

Like with external constraints, Oracle AI suffers from the problem that there would always be an incentive to create an AGI that could act on its own, without humans in the loop. Such an AGI would be far more effective in furthering whatever goals it had been built to pursue, but also far more dangerous.

Current-day narrow-AI technology includes high-frequency trading (HFT) algorithms, which make trading decisions within fractions of a second, far too fast to keep humans in the loop. HFT seeks to make a very short-term profit, but even traders looking for a longer-term investment benefit from being faster than their competitors. Market prices are also very effective at incorporating various sources of knowledge (Hanson 2007). As a consequence, a trading algorithm’s performance might be improved both by making it faster and by making it more capable of integrating various sources of knowledge. Since trading is also one of the fields with the most money involved, it seems like a reasonable presumption that most advances towards general AGI will quickly be put to use into making more money on the financial markets, with little opportunity for a human to vet all the decisions. Oracle AIs are unlikely to remain as pure oracles for long.

In general, any broad domain involving high stakes, adversarial decision-making, and a need to act rapidly is likely to become increasingly dominated by autonomous systems. The extent to which the systems will need general intelligence will depend on the domain, but many domains such as warfare, information security and fraud detection could plausibly make use of all the intelligence they can get. This is especially the case if one’s opponents in the domain are also using increasingly autonomous A(G)I, leading to an arms race where one might have little choice but to give increasing amounts of control to A(G)I systems.

From the same text, also related to Eliezer's points:

Even if humans were technically kept in the loop, they might not have the time, opportunity, or motivation to verify the advice given by an Oracle AI. This may be a danger even with more narrow-AI systems. Friedman & Kahn (1992) discuss this risk in the context of APACHE, a computer expert system that provides doctors with advice regarding treatments. They write that as the medical community starts to trust APACHE, it may become practice to act on APACHE’s recommendations somewhat automatically, and it may become increasingly difficult to challenge the “authority” of the recommendation. Eventually, the consultation system may in effect begin to dictate clinical decisions.

Likewise, Bostrom & Yudkowsky (2011) point out that modern bureaucrats often follow established procedures to the letter, rather than exercising their own judgment and allowing themselves to be blamed for any mistakes that follow. Dutifully following all the recommendations of an AGI system would be an even better way of avoiding blame.

Thus, even AGI systems that function purely to provide advice will need to be explicitly designed as safe in the sense of not providing advice that would go against human values (Wallach & Allen 2009). This requires a way of teaching them the correct values.

Comment author: torekp 15 June 2012 02:05:42AM 0 points [-]

I suspect this is the biggest counter-argument for Tool AI, even bigger than all the technical concerns Eliezer made in the post. Even if we could build a safe Tool AI, somebody would soon build an agent AI anyway.

Thank you for saying this (and backing it up better than I would have). I think we should concede, however, that a similar threat applies to FAI. The arms race phenomenon may create uFAI before FAI can be ready. This strikes me as very probable. Alternately, if AI does not "foom", uFAI might be created after FAI. (I'm mostly persuaded that it will foom, but I still think it's useful to map the debate.) The one advantage is that if Friendly Agent AI comes first and fooms, the threat is neutralized; whereas Friendly Tool AI can only advise us how to stop reckless AI researchers. If reckless agent AIs act more rapidly than we can respond, the Tool AI won't save us.

Comment author: Vladimir_Nesov 03 July 2012 09:53:31AM 1 point [-]

Alternately, if AI does not "foom", uFAI might be created after FAI.

If uFAI doesn't "foom" either, they both get a good chunk of expected utility. FAI doesn't need any particular capability, it only has to be competitive with other possible things.

Comment author: AndyWood 29 June 2012 08:00:34AM *  5 points [-]

Folks seem to habitually misrepresent the nature of modern software by focusing on a narrow slice of it. Google Maps is so much more than the pictures and text we touch and read on a screen.

Google Maps is the software. It is also the infrastructure running and delivering the software. It is the traffic sensors and cameras feeding it real-world input. Google Maps is also the continually shifting organization of brilliant human beings within Google focusing their own minds and each other's minds on refining the software to better meet users' needs and designers' intentions. It is the click data collected and aggregated to inform changes based on usage patterns. It is the GIS data and the collective efforts and intentions of everybody who collects GIS data or plans the collection thereof. It is the user-generated locale content and the collective efforts of everyone contributing that data.

To think of modern distributed software as merely a tool is to compartmentalize in the extreme. It is more like a many-way continuously evolving conversation among those creating it, between those creating it and those using it, and among those using it - plus the "conversation" from all the sensors, cameras, robots, cars, drivers, planes, pilots, computers, programmers, and everything else feeding the system data, both real-time and slow-changing. Whether the total system is "an agent" seems like a meaningless distinction to me. The system is already a continually evolving sum of the collective, purposeful action of everybody and everything who creates and interacts with Google Maps.

And that's just one web service among thousands in a world where the web services interact with each other, the companies and individuals behind them interact with each other, and so on. Arguing about the nature of the thingy on the phone or the monitor does not make any sense to me in light of the 100,000' view of the whole system.

Comment author: Nick_Beckstead 12 June 2012 05:29:08PM 5 points [-]

Eliezer argued that looking at modern software does not support Holden's claim that powerful tool AI is likely to come before dangerous agent AI. I'm not sure I think the examples he gave support his claim, especially if we broaden the "tool" concept in a way that seems consistent with Holden's arguments. I'm not to sure about this, but I would like to hear reactions.

Eliezer:

At one point in his conversation with Tallinn, Holden argues that AI will inevitably be developed along planning-Oracle lines, because making suggestions to humans is the natural course that most software takes. Searching for counterexamples instead of positive examples makes it clear that most lines of code don't do this. Your computer, when it reallocates RAM, doesn't pop up a button asking you if it's okay to reallocate RAM in such-and-such a fashion. Your car doesn't pop up a suggestion when it wants to change the fuel mix or apply dynamic stability control. Factory robots don't operate as human-worn bracelets whose blinking lights suggest motion. High-frequency trading programs execute stock orders on a microsecond timescale.

Whether this kind of software counts as agent-like software or tool software depends on what we mean by "tool." Holden glosses the distinction as follows:

In short, Google Maps is not an agent, taking actions in order to maximize a utility parameter. It is a tool, generating information and then displaying it in a user-friendly manner for me to consider, use and export or discard as I wish.

Defined in this way, it seems that most of this software is neither agent-like software nor tool software. I suggested an alternative definition in another comment:

An agent models the world and selects actions in a way that depends on what its modeling says will happen if it selects a given action. A tool may model the world, and may select actions depending on its modeling, but may not select actions in a way that depends on what its modeling says will happen if it selects a given action.

In this sense, I think all of Eliezer's examples of software is tool-like rather than agent-like (qualification: I don't know enough about the high-frequency trading stuff to say whether this is true there as well). I don't see these examples as strong support for the view that agent-like AGI is the default outcome.

More Eliezer:

Software that does happen to interface with humans is selectively visible and salient to humans, especially the tiny part of the software that does the interfacing; but this is a special case of a general cost/benefit tradeoff which, more often than not, turns out to swing the other way, because human advice is either too costly or doesn't provide enough benefit. Modern AI programmers are generally more interested in e.g. pushing the technological envelope to allow self-driving cars than to "just" do Google Maps.

It's clearly right that software does a lot of things without getting explicit human approval, and there are control/efficiency tradeoffs that explain why this is so. However, I suspect that the self-driving cars are also not agents in Holden's definition, or the one I proposed, and don't give a lot of support to the view that AGI will be agent-like. All this should be taken since a grain of salt since I don't too much about these cars. But I'm imagining these cars work by having a human select a place to go to, and then displaying a route, having the human accept the route, and then following a narrow set of rules to get the human there (e.g., stop if there's a red light such and such distance in front of you, brake if there's an object meeting such and such characteristics in your trajectory, etc.). I think the crucial thing here is the step where the human gets a helpful summary and then approves. That seems to fit my expansion of the "tool" concept, and seems to fit Holden's picture in the most important way: this car isn't going to do anything too crazy without our permission.

However, I can see an argument that advanced versions of this software would be changed to be more agent-like, in order to handle cases where the software has to decide what to do in split second situations that couldn't have easily been described in advance, such as whether to make some emergency maneuver to avoid an infrequent sort of collision. Perhaps examples of this kind would become more abundant if we thought about it; high frequency trading sounds like a good potential case for this.

Branches of AI that invoke human aid, like hybrid chess-playing algorithms designed to incorporate human advice, are a field of study; but they're the exception rather than the rule, and occur primarily where AIs can't yet do something humans do, e.g. humans acting as oracles for theorem-provers, where the humans suggest a route to a proof and the AI actually follows that route.

Quick thought: If it's hard to get AGIs to generate plans that people like, then it would seem that AGIs fall into this exception class, since in that case humans can do a better job of telling whether they like a given plan.

Comment author: Kawoomba 09 August 2012 02:20:53AM 13 points [-]

Marcus Hutter is a rare exception who specified his AGI in such unambiguous mathematical terms that he actually succeeded at realizing, after some discussion with SIAI personnel, that AIXI would kill off its users and seize control of its reward button.

Marcus Hutter denies ever having said that.

I asked EY for how to proceed, with his approval these are the messages we exchanged:

Eliezer,

I am unsure how to proceed and would appreciate your thoughts on resolving this situation:

In your Reply to Holden on 'Tool AI', to me one of the central points, and the one that much credibility hinges on is this:

[Initial quote of this comment]

That and some other "quotes" and allusions to Hutter, the most recent one by Carl Shulman [I referred to this: "The informal argument that AIXI would accept a delusion box to give itself maximal sensory reward was made by Eliezer a while ago, and convinced the AIXI originators." which I may have mistakingly attributed to M.H. since he is the AIXI originator], that are attributed to M.H. seemed to be greatly at odds with my experiences with the man, so I asked Carl Shulman for sourcing them, he had this to say:

"I recall overhearing part of a conversation at the Singularity Summit a few years ago between Eliezer and Schmidhuber, with pushback followed later by agreement. It may have been initial misunderstanding, but it looked consistent with Eliezer's story."

At this point I asked Marcus himself, whom I know peripherally, for general clarification.

Marcus linked to the relevant sentence quoted above from your Reply to Holden and stated in unambiguous terms that he never said that. Further, he stated that while he does not have the time to engage in such discussions, he authorised me to set the picture straight.

I'm sure you realize what this seems to look like (note: not a harmless misunderstanding, though that is possible).

[Redacted personal info] ... and though I currently would not donate to your specific cause, I also am unsure about causing the potential ramifications of you quoting Hutter for support wrongly. On the other hand, I don't feel like a silent edit would do justice to the situation.

If you have a constructive idea of how to settle this issue, please let me know.

EY's response:

Info like this should be posted, and you can quote me on that part too. I did notice a tendency of Marcus Hutter to unconvince himself of things and require reconvincing, and the known experimental fragility of human memory (people believe they have always believed their conclusions, more or less by default) suggests that this is an adequate explanation for everything, especially if Carl Shulman remembers a similar conversation from his viewpoint. It is also obviously possible that I have misremembered and then caused false memory in Carl. I do seem to recall pretty strongly that Hutter invented the Azathoth Equation (for an AIXI variant with an extremely high exponential discount, so it stays in its box pressing its button so long as it doesn't anticipate being disturbed in the next 5 seconds) in response to this acknowledged concern, and I would be surprised if Hutter doesn't remember the actual equation-proposal. My ideal resolution would be for Hutter and I to start over with no harm, no foul on both sides and do a Bloggingheads about it so that there's an accessible record of the resulting dialogue. Please feel free to post your entire comment along with this entire response.

I apologise for the confusion of Carl Shulman actually referring to overhearing a conversation with Schmidhuber (again, since his initial quote referred to just "AIXI originators" I pattern matched that to M.H.), so disregard EY's remark on potentially causing false memories on Carl Shulman's part.

However, the main point of M.H. contradicting what is attributed to him in the Reply to Holden on 'Tool AI' stands.

For full reference, linking the relevant part of M.H.'s email:

[This part is translated, thus paraphrased:] I don't have time to participate in blog discussions, I do know there's a quote of me floating around: [Link to initial quote of this comment along with its text]

[Other than incorporating the links he provided into the Markdown Syntax, this following part is verbatim:]

I never said that. These are mainly open questions.

See e.g. Sec.5 of One Decade of Universal Artificial Intelligence In Theoretical Foundations of Artificial General Intelligence (2012) 67?--88? and references therein (in particular to Laurent Orseau) for social questions regarding AIXI.

See also Can Intelligence Explode? Journal of Consciousness Studies, 19:1-2 (2012) 143-166 for a discussion of AIXI in relation to the Singularity.

I also recommend you subscribe to the Mathematical Artificial General Intelligence Consortium (MAGIC) mailing list for a more scientific discussion on these and related issues.

Before taking any more of his time, and since he does not agree with the initial quote (at least now, whether he did back then is in dispute), I suggest the "Reply to Holden on Tool AI" to reflect that. Further, I suggest to instead refer to the sources he gave for a more thorough examination on his views re: AIXI.

Comment author: lukeprog 01 May 2013 06:01:35PM *  3 points [-]

I don't know whether Hutter ever told Eliezer that "AIXI would kill off its users and seize control of its reward button," but he does say the following in his book (pp. 238-239):

Another problem connected, but possibly not limited to embodied agents, especially if they are rewarded by humans, is the following: Sufficiently intelligent agents may increase their rewards by psychologically manipulating their human "teachers", or by threatening them... Every intelligence superior to humans is capable of manipulating the latter. In the absence of manipulable humans, e.g. where the reward structure serves a survival function, AIXI may directly hack into its reward feedback. Since this is unlikely to increase its long-term survival, AIXI will probably resist this kind of manipulation (just as most humans don't take hard drugs, because of their long-term catastrophic consequences).

This issue is discussed at greater length, and with greater formality, in Dewey (2011) and Ring & Orseau (2011).

Comment author: MBlume 12 June 2012 11:40:44PM 12 points [-]

I begin by thanking Holden Karnofsky of Givewell for his rare gift of his detailed, engaged, and helpfully-meant critical article Thoughts on the Singularity Institute (SI). In this reply I will engage with only one of the many subjects raised therein, the topic of, as I would term them, non-self-modifying planning Oracles, a.k.a. 'Google Maps AGI' a.k.a. 'tool AI', this being the topic that requires me personally to answer. I hope that my reply will be accepted as addressing the most important central points, though I did not have time to explore every avenue. I certainly do not wish to be logically rude, and if I have failed, please remember with compassion that it's not always obvious to one person what another person will think was the central point.

Luke Mueulhauser and Carl Shulman contributed to this article, but the final edit was my own, likewise any flaws.

I think you're starting to write more like a Friendly AI. This is totally a good thing.

Comment author: John_Maxwell_IV 13 June 2012 05:58:48AM 7 points [-]

Yes, the tone of this response should be commended.

Comment author: private_messaging 13 June 2012 07:32:39AM 2 points [-]

Wouldn't even a paperclip maximizer write in same style in those circumstances?

Comment author: John_Maxwell_IV 14 June 2012 01:20:51AM 2 points [-]

IMO, speaking in arrogant absolutes makes people stupid regardless of what conclusion you're arguing for.

Comment author: wedrifid 13 June 2012 09:58:22AM 0 points [-]

Wouldn't even a paperclip maximizer write in same style in those circumstances?

No. It would start hacking things, take over the world, kill everything then burn the cosmic commons.

Comment author: private_messaging 13 June 2012 10:57:15AM 3 points [-]

Only when it has power to do that. Meatbound equivalent would have to upload itself first.

Comment author: MarkusRamikin 13 June 2012 07:54:41AM 1 point [-]

Maybe that was Luke's contribution ;)

Comment author: futo 14 June 2012 06:40:54AM 4 points [-]

Is Google Maps such a good example of a tool AI?

If a significant amount of people is using google maps to decide their route, then solving queries from multiple users while coordinating the responses to each request is going to provide a strong advantage in terms of its optimization goal and will probably be an obvious feature to implement. The responses from the tool are going to be shaping the city traffic.

If this is the case, It's going to be extremely hard for humans to supervise the set of answers given by google maps (Of course, individual answers are going to be read by the end users, but that will be provide no insight on what it is really doing at a high level).

Having our example AI deciding where a lot of people is going to be at different times based on some optimization function looks really close to the idea of an agent AI directly acting on our world.

Comment author: [deleted] 12 June 2012 01:20:59PM 4 points [-]

Writing nitpick:

It's sort of like thinking that a machine learning professional who did sales optimization for an orange company couldn't possibly do sales optimization for a banana company, because their skills must be about oranges rather than bananas.

This is a terrible analogy. It assumes what you're trying to prove, oversimplifies a complex issue, and isn't even all that analogous to the issue at hand. Sales optimization for a banana company is obviously related to sales optimization in an orange company; not so with Oracle Al and Friendly AI.

Comment author: thomblake 12 June 2012 02:12:15PM *  1 point [-]

Sales optimization for a banana company is obviously related to sales optimization in an orange company; not so with Oracle Al and Friendly AI.

The goal with an analogy is to have the reader see the connection as obvious in the analogous case. It's not a flaw.

Comment author: [deleted] 12 June 2012 02:48:46PM 1 point [-]

Yes, but the analogy is a drastic oversimplification of Oracle/FAI case, and it assumes the conclusion it is supposed to be demonstrating.

Comment author: thomblake 12 June 2012 03:03:08PM *  5 points [-]

I don't see how it assumes what it's trying to prove. The analogous case is not about the relationship between Oracle AI and Friendly AI. For A:B::C:D to be a good analogy, C:D should have the same relationship that you're asserting A:B has, and A:B should be relevantly similar to C:D, and A,B,C, and D should all be different things. You can argue that it fails at one or several of those, but it really isn't begging the question unless you end up with something like A:B::A:B.

An analogy should be a simplification. In using an analogy, one is assuming the reader is not sufficiently versed in the complexities of A:B but will see the obviousness of C:D.

Comment author: [deleted] 12 June 2012 03:36:30PM *  2 points [-]

Thank you for putting it in such clear language. In this case, C and D (banana sales and orange sales) are defined to be obviously identical, even to the layperson. To claim A:B::C:D is a drastic oversimplification of the actual relationship between A and B, a relationship that has a number of properties that the relationship between C and D does not have. Moreover, the analogy does not demonstrate why A:B::C:D, it simply asserts that it would be oh-so-obvious to anyone that D is identical to C and then claims that the case of A and B is the same. Consequently, the analogy is used as an assertion, a way of insisting A:B to the reader rather than demonstrating why it is so.

Comment author: JGWeissman 12 June 2012 04:49:54PM 1 point [-]

The analogy on its own is just an assertion. That assertion is backed up by detailed points in the rest of the article demonstrating the asserted similarities, like the required skills of looking at a mathematical specification of a program and predicting how that program will really behave, finding methods of choosing actions/plans that are less expensive than searching the entire solution space but still return a result high in the preference order, and specifying the preference order to actually reflect what we want.

Comment author: shminux 15 June 2012 09:12:27PM *  10 points [-]

Probably nothing new, but I just wanted to note that when you couple two straightforward Google tools, Maps and a large enough fleet of self-driving cars, they are likely to unintentionally agentize by shaping the traffic.

For example, the goal of each is to optimize the fuel economy/driving time, so the routes Google cars would take depend on the expected traffic volume, as predicted by Maps access, among other things. Similarly, Maps would know where these cars are or will be at a given time, and would adjust its output accordingly (possibly as a user option). An optimization strategy might easy arise that gives Google cars preference over other cars, in order to minimize, say, the overall emission levels. This can be easily seen as unfriendly by a regular Map user, but friendly by the municipality.

Similar scenarios would pop up in many cases where, in the EE speak, a tool gains an intentional or a parasitic feedback, whether positive or negative. As anyone who dealt with music amps knows, this feedback appears spontaneously and is often very difficult to track down. In a sense, a tool as simple as an amp can agentize and drown the positive signal. As the tool complexity grows, so do the odds of parasitic feedback. Coupling multiple "safe" tools together increases such odds exponentially.

Comment author: JGWeissman 15 June 2012 10:20:36PM 0 points [-]

Google maps finds routes for individual users that rank high in the preference ordering specified by minimizing distance, expected time given traffic, or some other simple metric. The process for finding the route for any particular individual is isolated from the process for finding the route for other users; the tool does not consider the effect of giving a route to user A on the driving time of user B. Such a system is possible to design and implement, but merely giving Google maps data of where a particular class of users are driving in real time, and having those users request routes in real time, does not change what algorithm Google maps will use to suggest routes, even if another algorithm would help it better optimize driving time, the purpose for which its current algorithm was programmed. Google maps is not meta enough to explore alternate optimization strategies.

(And if the sufficiently meta human engineers at Google were to implement such a system, in which other users were systematically instructed to make sacrifices for the benifet of Google cars, the other users would switch to other mapping and routing providers.)

Comment author: shminux 15 June 2012 10:37:27PM *  1 point [-]

in which other users were systematically instructed to make sacrifices for the benifet of Google cars

I agree, but this is only one possible scenario. It is also likely that a fleet of Google cars would benefit the overall traffic patterns by routing them away from congested areas. In such a way, even giving priority to Google cars might provide an overall benefit to regular drivers, due to reduced congestion.

In any case, my point was less about the current implementation of Google Maps and more about the possibility that combining tools can lead to parasitic agentization.

Comment author: wedrifid 12 June 2012 06:08:04AM 9 points [-]

This is the first time I can recall Eliezer giving an overt indication regarding how likely an AGI project is to doom us. He suggests that 90% chance of Doom given intelligent effort is unrealistically high. Previously I had only seem him declare that FAI is worth attempting once you multiply. While he still hasn't given numbers (not saying he should) he has has given a bound. Interesting. And perhaps a little more optimistic than I expected - or at least more optimistic than I would have expected prior to Luke's comment.

Comment author: Lightwave 12 June 2012 09:56:04AM *  9 points [-]

how likely an AGI project is to doom us

Isn't it more like "how likely a formally proven FAI design is to doom us", since this is what Holden seems to be arguing (see his quote below)?

Suppose that it is successful in the "AGI" part of its goal, i.e., it has successfully created an intelligence vastly superior to human intelligence and extraordinarily powerful from our perspective. Suppose that it has also done its best on the "Friendly" part of the goal: it has developed a formal argument for why its AGI's utility function will be Friendly, it believes this argument to be airtight, and it has had this argument checked over by 100 of the world's most intelligent and relevantly experienced people. .. What will be the outcome?

Comment author: JamesAndrix 16 June 2012 10:47:44PM 1 point [-]

"When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong."

http://en.wikipedia.org/wiki/Clarke%27s_three_laws

Comment author: gwern 04 August 2012 06:52:59PM *  6 points [-]

I think it's a pity that we're not focusing on what we could do to test the tool vs general AI distinction. For example, here's one near-future test: how do we humans deal with drones?

Drones are exploding in popularity, are increasing their capabilities constantly, and are coveted by countless security agencies and private groups for their tremendous use in all sorts of roles both benign and disturbing. Just like AIs would be. The tool vs general AI distinction maps very nicely onto drones as well: a tool AI corresponds to a drone being manually flown by a human pilot somewhere, while a general AI would correspond to an autonomous drone which is carrying out some mission (blast insurgents?).

So, here is a near-future test of the question 'are people likely to let tool AIs 'drive themselves' for greater efficiency?' - simply ask whether in, say, a decade there are autonomous drones carrying tasks that now would only be carried out by piloted drones.

If in a decade we learn that autonomous drones are killing people, then we have an answer to our tool AI question: it doesn't matter because given a tool AI, people will just turn it into a general AI.

(Amdahl's law: if the human in the loop takes up 10% of the time, and the AI or drone part comprises the other 90%, then even if the drone or AI become infinitely fast, you will still never speed up the whole loop by more than 90%... until you hand over that 10% to the AI, that is. EDIT: See also https://web.archive.org/web/20121122150219/http://lesswrong.com/lw/f53/now_i_appreciate_agency/7q4o )

Comment author: gwern 04 September 2014 06:09:46PM 4 points [-]

Besides Knight Capital, HFT may provide another example of near-disaster from economic incentives forcing the removal of safety guidelines from narrow AI. From the LRB's "Be grateful for drizzle: Donald MacKenzie on high-frequency trading":

Such events don’t always become public. In a New York coffeehouse, a former high-frequency trader told me matter of factly that one of his colleagues had once made the simplest of slip-ups in a program: what mathematicians call a ‘sign error’, interchanging a plus and a minus. When the program started to run it behaved rather like the Knight program, building bigger and bigger trading positions, in this case at an exponential rate: doubling them, then redoubling them, and so on. ‘It took him 52 seconds to realise what was happening, something was terribly wrong, and he pressed the red button,’ stopping the program. ‘By then we had lost $3 million.’ The trader’s manager calculated ‘that in another twenty seconds at the rate of the geometric progression,’ the trading firm would have been bankrupt, ‘and in another fifty or so seconds, our clearing broker’ – a major Wall Street investment bank – ‘would have been bankrupt, because of course if we’re bankrupt our clearing broker is responsible for our debts … it wouldn’t have been too many seconds after that the whole market would have gone.’

What is most telling about that story is that not long previously it couldn’t have happened. High-frequency firms are sharply aware of the risks of bugs in programs, and at one time my informant’s firm used an automated check that would have stopped the errant program well before its human user spotted that anything was wrong. However, the firm had been losing out in the speed race, so had launched what my informant called ‘a war on latency’, trying to remove all detectable sources of delay. Unfortunately, the risk check had been one of those sources.

(Memoirs from US drone operators suggest that the bureaucratic organizations in charge of racking up kill-counts have become disturbingly cavalier about not doing their homework on the targets they're blowing up, but thus far, anyway, they haven't made the drones fully autonomous.)

Comment author: Stuart_Armstrong 14 June 2012 05:30:35PM 6 points [-]

Minor point from Nick Bostrom: an agent AI may be safer than a tool AI, because if something goes unexpectedly wrong, then an agent with safe goals should turn out to be better than a non-agent whose behaviour would be unpredictable.

Comment author: TheOtherDave 14 June 2012 06:11:30PM 10 points [-]

Also, an agent with safer goals than humans have (which is a high bar, but not nearly as high a bar as some alternatives) is safer than humans with equivalently powerful tools.

Comment author: David_Gerard 08 August 2012 09:01:32AM 5 points [-]

Marcus Hutter is a rare exception who specified his AGI in such unambiguous mathematical terms that he actually succeeded at realizing, after some discussion with SIAI personnel, that AIXI would kill off its users and seize control of its reward button.

A couple of people have enquired with Hutter and he has denied saying this. So it appears a citation is needed.

Comment author: Eliezer_Yudkowsky 08 August 2012 06:46:07PM 7 points [-]

I'll try to get the results in writing the next time we have a discussion. Human memory is a fragile thing under the best of circumstances.

Comment author: HBDfan 09 August 2012 08:08:09AM *  0 points [-]

[delete]

Comment author: Vaniver 12 June 2012 05:12:08AM 9 points [-]

Commentary (there will be a lot of "to me"s because I have been a bystander to this exchange so far):

I think this post misunderstands Holden's point, because it looks like it's still talking about agents. Tool AI, to me, is a decision support system: I tell Google Maps where I will start from and where I will leave from, and it generates a route using its algorithm. Similarly, I could tell Dr. Watson my medical data, and it will supply a diagnosis and a treatment plan that has a high score based on the utility function I provide.

In neither case are the skills of "looking at the equations and determining real-world consequences" that necessary. There are no dark secrets lurking in the soul of A*. Indeed, that might be the heart of the issue: tool AI might be those situations where you can make a network that represents the world, identify two nodes, and call your optimization algorithm of choice to determine the best actions to choose to attempt to make it from the start node to the end node.

Reducing the world to a network is really hard. Determining preferences between outcomes is hard. But Tool AI looks to me like saying "well, the whole world is really too much. I'm just going to deal with planning routes, which is a simple world that I can understand," where the FAI tools aren't that relevant. The network might be out of line with reality, the optimization algorithm might be buggy or clumsy, but the horror stories that keep FAI researchers up at night seem impossible because of the inherently limited scope, and the ability to do dry runs and simulations until the AI's model of reality is trusted enough to give it control.

Now, this requires that AI only be used for things like planning where to put products on shelves, not planning corporate strategy- but if you work from the current stuff up rather than from the God algorithm down, it doesn't look like corporate strategy will be on the table until AI is developed to the point where it could be trusted with that. If someone gave me a black box that spit out plans based on English input, then I wouldn't trust it and I imagine you wouldn't either- but I don't think that's what we're looking at, and I don't know if planning for that scenario is valuable.

It seems to me that SI has discussed Holden's Tool AI idea- when it made the distinction between AI and AGI. Holden seems to me to be asking "well, if AGI is such a tough problem, why even do it?".

Comment author: Kaj_Sotala 12 June 2012 09:19:13PM 8 points [-]

Holden explicitly said that he was talking about AGI in his dialogue with Jaan Tallinn:

Jaan: so GMAGI would -- effectively -- still be a narrow AI that's designed to augment human capabilities in particularly strategic domains, while not being able to perform tasks such as programming. also, importantly, such GMAGI would not be able to make non-statistical (ie, individual) predictions about the behaviour of human beings, since it is unable to predict their actions in domains where it is inferior.

Holden: [...] I don't think of the GMAGI I'm describing as necessarily narrow - just as being such that assigning it to improve its own prediction algorithm is less productive than assigning it directly to figuring out the questions the programmer wants (like "how do I develop superweapons"). There are many ways this could be the case.

Jaan: [...] i stand corrected re the GMAGI definition -- from now on let's assume that it is a full blown AGI in the sense that it can perform every intellectual task better than the best of human teams, including programming itself.

Comment author: Dr_Manhattan 12 June 2012 12:29:31PM 5 points [-]

I think you're arguing about Karnovsky's intention, but it seems clear (to me :) that he is proposing something much more general that a strategy of pursuing best narrow AIs - see the "Here's how I picture the Google Maps AGI " code snipped Eliezer is working of.

In any case, taking your interpretation as your proposal, I don't think anyone is disagreeing with the value of building good narrow AIs where we can, the issue is that the world might be economically driven towards AGI, and someone needs to do the safety research, which is essentially the SI mission.

Comment author: thomblake 12 June 2012 02:05:31PM 4 points [-]

I don't think the distinction is supposed to be merely the distinction between Narrow AI and AGI. The "tool AI" oracle is still supposed to be a general AI that can solve many varied sorts of problems, especially important problems like existential risk.

And it doesn't make sense to "propose" Narrow AI - we have plenty of that already, and nobody around here seems to be proposing that we stop that.

Comment author: Daermonn 12 June 2012 07:49:35AM 3 points [-]

This really gets at the heart of what intuitively struck me wrong (read: "confused me") in Eliezer's reply. Both Eliezer and Holden engage with the example "Google Maps AGI"; I'm not sure what the difference is - if any - between "Google Maps AGI" and the sort of search/decision-support algorithms that Google Maps and other GPS systems currently use. The algorithm Holdon describes and the neat A* algorithm Eliezer presents seem to just do exactly what the GPS on my phone already does. If the Tool AI we're discussing is different than current GPS systems, then what is the difference? Near as I understand it, AGI is intelligent across different domains in the same way a human is, while Tool AI (= narrow AI?) is the sort of simple-domain search algorithms we see in GPS. Am I missing something here?

But if what Holden is talking about by Tool AI is just this sort of simple(r), non-reflective search algorithm, then I understand why he thinks this is significantly less risky; GPS-style Tool AI only gets me lost when it screws up, instead of killing the whole human species. Sure, this tool is imperfect: sometimes it doesn't match my utility function, and returns a route that leads me into traffic, or would take too long, or whatever; sometimes it doesn't correctly model what's actually going on, and thinks I'm on the wrong street. Even still, gradually building increasingly agentful Tool AIs - ones that take more of the optimization process away from the human user - seems like it would be much safer than just swinging for the fences right away.

So I think that Vaniver is right when he says that the heart of Holden's Tool AI point is "Well, if AGI is such a tough problem, why even do it?"

This being said, I still think that Eliezer's reply succeeds. I think his most important point is the one about specialization: AGI and Tool AI demand domain expertise to evaluate arguments about safety, and the best way to cultivate that expertise is with an organization that specializes in FAI-grade programmers. The analogy with the sort of optimal-charity work Holden specializes in was particularly weighty.

I see Eliezer's response to Holden's challenge - "why do AGI at all?" - as: "Because you need FAI-grade skills to know if you need to do AGI or not." If AGI is an existential threat, and you need FAI-grade skills to know how to deal with that threat, then you need FAI-grade programmers.

(Though, I don't know if "The world needs FAI-grade programmers, even if we just want to do Tool AI right now" carries through to "Invest in SIAI as a charity," which is what Holden is ultimately interested in.)

Comment author: Viliam_Bur 12 June 2012 05:05:35PM 6 points [-]

Let's be a bit more specific -- that is one important point of the article, that as soon as the "Tool AI" definition becomes more specific, the problems start to appear.

We don't want just a system that finds a route between points A and B. We have Google Maps already. By speaking about AGI we want a system that can answer "any question". (Not literally, but it means a wide range of possible question types.) So we don't need an algorithm to find the shortest way between A and B, but we need an algorithm to answer "any question" (or admit that it cannot find an answer), and of course to answer that question correctly.

So could you be just a bit more specific about the algorithm that provides a correct answer to any question? ("I don't know" is also a correct answer, if the system does not know.) Because that is the moment when the problems become visible.

Don't talk about what the Tool AI doesn't do, say what it does. And with a high probability there will be a problem. Of course until you tell what exactly the Tool AI will do, I can't tell you how exactly that problem will happen.

This is relevant:

Marcus Hutter is a rare exception who specified his AGI in such unambiguous mathematical terms that he actually succeeded at realizing, after some discussion with SIAI personnel, that AIXI would kill off its users and seize control of its reward button.

Please note that AIXI with outputs connected only to a monitor seems like an instance of the Tool AI.

Comment author: Nick_Beckstead 12 June 2012 05:43:57PM 1 point [-]

Please note that AIXI with outputs connected only to a monitor seems like an instance of the Tool AI.

As I read Holden, and on my proposed way of making "agent" precise, this would be an agent rather than a tool. The crucial thing is that this version of AIXI selects actions on the basis of how well they serve certain goals without user approval. If you had a variation on AIXI that identified the action that would maximize a utility function and displayed the action to a user (where the method of display was not done in an open-ended goal-directed way), that would count as a tool.

Comment author: TruePath 03 August 2012 03:40:34PM 2 points [-]

I'm deeply confused. How can you even define the difference between tool AI and FAI?

I assume that even tool AI is supposed to be able to opine on relatively long sequences of input. In particular, to be useful it must be able to accumulate information over essentially unbounded time periods. Say if you want advise about where to position your air defenses you must be able to go back to the AI system each day hand it updates on enemy activity and expect it to integrate that information with information it received during previous sessions. Whether or not you upload this info each time you ask a quesiton or not in effect the AI has (periods) in which it is loaded with a significant amount of information about past events.

But now you face the problem that self-modification is indistinguishable from simple storing of data. The existence of universal Turing machines demonstrate that much. Simply by loading up information in memory one can generate behavior corresponding to any kind of (software) self-modification.

So perhaps the supposed difference is that this AI won't actually take direct actions, merely make verbal suggestions. Well it's awful optimistic to suppose no one will get lazy or exigencies won't drive them to connect a simple script up to the machine which takes say sentences of the form "I recommend you deploy your troops in this manner." and directly sends the orders. Even if so the machine still takes direct action in the form of making statements that influence human behavior.

You might argue that a tool AI is one in which the advice it generates doesn't require self-reference or consideration of it's future actions so it is somehow different in kind. However, again simple analysis reveals this can't be so. Imagine again the basic question of "How should I position my forces to defend against the enemy attack." Now, given that the enemy is likely to react in certain ways correct advice requires the tool AI to consider whether future responses will be orchestrated by itself or a human who will be unable to handle certain kinds of complexity or be inclined to different sorts of responses. Those even a purely advisory AI needs the ability to project likely outcomes based on it's on likely future behaviors.

Now it seems we are again in the realm of 'FAI' since one has to ensure that the advice given by the machine when presented with indefinitely long, complex historical records won't end up encouraging the outcome where someone ends up connecting permanent memory and wiring on the ability to take direct action. After all, if the advise is designed to be of maximum usefulness to the people asking the tool AI must be programmed to give advice that causes them to best achieve the goals they ask for advice in achieving. Since such goals could quite reasonably be advanced by the ability of the AI to take direct action and the reasons for the advice can't ever be entirely explained to humans (even deep blue goes beyond being able to do that to humans now) I don't see how the problem isn't just as complicated as 'FAI'.

I guess it comes down to my belief that if you can't formulate the notion precisely I'm skeptical it's coherent.

Comment author: Jonathan_Graehl 12 June 2012 09:40:10PM *  2 points [-]

I was briefly tripped up by the use of "risk gradient between X and Y" to indicate how much riskier X is than Y (perhaps "gradient" evokes a continuum between X and Y). I'd strike the jargon, or explain what it means.

"Holden should respect our difficult-to-explain expertise just as we ask others to respect Holden's" might actually be persuasive to Holden (smart people often forget to search for ideas via an empathic perspective), but it's whiny as a public signal.

Comment author: JGWeissman 12 June 2012 10:13:42PM 2 points [-]

"Holden should respect our difficult-to-explain expertise just as we ask others to respect Holden's"

That is not an actual quote, and I think it misrepresents Eliezer's actual point, which is that the problem of FAI, like finance and philanthropy, involves pitfalls that you can fall into without even realizing it and it is worthwhile to have full time professionals learning how to avoid those pitfalls.

Comment author: Eliezer_Yudkowsky 13 June 2012 12:04:31AM 3 points [-]

...or at least full-time professionals who know that the pitfalls exist, so they can move forward if they learn to avoid pitfalls and otherwise take different routes.

Comment author: Jonathan_Graehl 14 June 2012 04:06:35AM *  0 points [-]

It's pretty deeply analogous (deeper than my "paraphrase" indicated), but I'm not sure it serves you well as part of any public response.

I found it convincing but off-putting.

Comment author: Jonathan_Graehl 14 June 2012 04:05:01AM 0 points [-]

Fair enough (I didn't mean to represent it as an exact gloss), but obviously my quoted paraphrase actually represents the meaning as I took it (or rather, some pattern-matching part of me that I wouldn't stand by, but feel comfortable projecting onto the "public").

Comment author: Sniffnoy 12 June 2012 10:20:20AM 2 points [-]

I'm surprised to see no mention of the old "How do you ensure that your Oracle AI doesn't scribble over the world in order to gain more computational resources with which to answer your question?" argument.

Comment author: PECOS-9 12 June 2012 05:33:46AM 2 points [-]

I think the link on Demis Hassabis in section 3 is incorrect . It is the same as the Ray Kurzweil link.

Comment author: HoldenKarnofsky 01 August 2012 02:09:11PM 3 points [-]

To summarize how I see the current state of the debate over "tool AI":

  • Eliezer and I have differing intuitions about the likely feasibility, safety and usefulness of the "tool" framework relative to the "Friendliness theory" framework, as laid out in this exchange. This relates mostly to Eliezer's point #2 in the original post. We are both trying to make predictions about a technology for which many of the details are unknown, and at this point I don't see a clear way forward for resolving our disagreements, though I did make one suggestion in that thread.
  • Eliezer has also made two arguments (#1 and #4 in the original post) that appear to be of the form, "Even if the 'tool' approach is most promising, the Singularity Institute still represents a strong giving opportunity." A couple of thoughts on this point:
    • One reason I find the "tool" approach relevant in the context of SI is that it resembles what I see as the traditional approach to software development. My view is that it is likely to be both safer and more efficient for developing AGI than the "Friendliness theory" approach. If this is the case, it seems that the safety of AGI will largely be a function of the competence and care with which its developers execute on the traditional approach to software development, and the potential value-added of a third-party team of "Friendliness specialists" is unclear.
    • That said, I recognize that SI has multiple conceptually possible paths to impact, including developing AGI itself and raising awareness of the risks of AGI. I believe that the more the case for SI revolves around activities like these rather than around developing "Friendliness theory," the higher the bar for SI's general impressiveness (as an organization and team) becomes; I will elaborate on this when I respond to Luke's response to me.
  • Regarding Eliezer's point #3 - I think this largely comes down to how strong one finds the argument for "tool A.I." I agree that one shouldn't expect SI to respond to every possible critique of its plans. But I think it's reasonable to expect it to anticipate and respond to the stronger possible critiques.
  • I'd also like to address two common objections to the "tool AI" framework that came up in comments, though neither of these objections appears to have been taken up in official SI responses.
    • Some have argued that the idea of "tool AI" is incoherent, or is not distinct from the idea of "Oracle AI," or is conceptually impossible. I believe these arguments to be incorrect, though my ability to formalize and clarify my intuitions on this point has been limited. For those interested in reading attempts to better clarify the concept of "tool AI" following my original post, I recommend jsalvatier's comments on the discussion post devoted to this topic as well as my exchange with Eliezer elsewhere on this thread.
    • Some have argued that "agents" are likely to be more efficient and powerful than "tools," since they are not bottlenecked by human input, and thus that the "tool" concept is unimportant. I anticipated this objection in my original post and expanded on my response in my exchange with Eliezer elsewhere on this thread. In a nutshell, I believe the "tool" framework is likely to be a faster and more efficient way of developing a capable and useful AGI than the sort of framework for which "Friendliness theory" would be relevant; and if it isn't, that the sort of work SI is doing on "Friendliness theory" is likely to be of little value. (Again, I recognize that SI has multiple conceptually possible paths to impact other than development of "Friendliness theory" and will address these in a future comment.)
Comment author: timtyler 03 August 2012 01:19:23AM -3 points [-]

Some have argued that "agents" are likely to be more efficient and powerful than "tools," since they are not bottlenecked by human input, and thus that the "tool" concept is unimportant.

Currently machines are enslaved by humans. It's a common delusion that we'll be able to keep them that way.

All plans start off with machines as tools. Only unrealistic plans have machines winding up as tools.

Comment author: Arepo 13 June 2012 11:47:51AM 2 points [-]

Seems like a decent reply overall, but I found the fourth point very unconvincing. Holden has said 'what he knows know' - to wit that whereas the world's best experts would normally test a complicated programme by running it, isolating out what (inevitably) went wrong by examining the results it produced, rewriting it, then doing it again.

Almost no programmes are glitch free, so this is at best an optimization process and one which - as Holden pointed out - you can't do with this type of AI. If (/when) it goes wrong the first time, you don't get a second chance. Eliezer's reply doesn't seem to address this stark difference between what experts have been achieving and what SIAI is asking them to achieve.

Comment author: bogdanb 10 July 2013 08:55:17PM *  0 points [-]

I agree with the glitch problems. But (1) programmers and techniques are improving; (2) people are more careful when aware of danger; (3) if it’s hard but inevitable, giving up doesn’t sound like a winning strategy. I mean, if people make mistakes at some important task, how isn’t it a good idea to get lots of smart mathematicians to think hard about how to avoid mistakes?

Note that all doctors, biologists, nuclear physicists and rocket scientists are also not glitch free, but those that work with dangerous stuff do tend to err less often. But they have to be aware of the dangers (or at least anticipate their existence). A doctor might try a different pill if the first one doesn’t seem to work well against the sniffles, but will be much less inclined to experiments when they know the problem is a potential pandemic.

(By the way, it is probably possible that the first possible AGI is buggy, and a killer, and will foom in a few seconds (or before anyone can react, anyway); it might even be likely. But it’s still possible we’ll get several chances. My point is not that we don’t have to worry about anything, but that even if the chances might be low it still makes sense to try harder. And, hey, AFAIK the automatic trains in Paris work much better than the human-driven ones. It’s not quite a fair comparison in any direction, but there is evidence that we can make stuff work pretty well at least for a while.)

ETA: You know, now that I think about it, it seems plausible that programmer errors would lean towards the AGI not working (e.g. you divide by zero; core dump; the program stops), while a mathematician’s error would lean towards the AGI working but doing something catastrophic (e.g. your encryption program has exactly zero bugs, it works exactly as designed, but ROT13 has been proven cryptographically unsound after you used it to send that important secret). So maybe it’s a good idea if the math guys start thinking hard long in advance?

Comment author: private_messaging 13 July 2012 03:27:21PM *  1 point [-]

Marcus Hutter is a rare exception who specified his AGI in such unambiguous mathematical terms that he actually succeeded at realizing, after some discussion with SIAI personnel, that AIXI would kill off its users and seize control of its reward button. But based on past sad experience with many other would-be designers, I say "Explain to a neutral judge how the math kills" and not "Explain to the person who invented that math and likes it."

Any sources to this extraordinary claim? Hutter's own statements? Cartesian-dualist AI has real trouble preserving itself against shut down, which you yourself have noted. It has to somehow have a model where reward disappears if it stops being computed, or you get the AI that would shut itself down when reward is pressed, and that's it. edit: I.e. it is pretty clear that AIXI is not a friendly AI and can kill you, that's pretty agreeable, but it remains to be shown that it would be hard to kill AIXI (assuming it can't do infinite recursion predicting itself).

edit2: and of course, nothing in AIXI fundamentally requires that you sum the reward over a "large number of future steps" rather than 1 step. (I don't think its scarier summing over unlimited number of steps though, think what sort of models it can make if it ever observes effects of slight temperature caused variations in the CPU clock rate for example, against the physics model it has on it's other input. If it can't understand speeding up itself, it'll figure it slows down entire universe, more rewards per external risk. Here's one anvil onto the head: overclocking, or just straight fan shutdown so that internal temperature rises and the quartz clock ticks a teeny bit faster. I think it is going to be deviously clever at killing itself as soon as possible. Hutter likes his math may be the reason why you can convince him it will actually be smart enough to kill people)

Comment author: Johnicholas 12 June 2012 10:23:35AM 1 point [-]

The thing that is most like an agent in the Tool AI scenario is not the computer and software that it is running. The agent is the combination of the human (which is of course very much like an agent) together with the computer-and-software that constitutes the tool. Holden's argument is that this combination agent is safer somehow. (Perhaps it is more familiar; we can judge intention of the human component with facial expression, for example.)

The claim that Tool AI is an obvious answer to the Friendly AI problem is a paper tiger that Eliezer demolished. However, there's a weaker claim, that SIAI is not thinking about Tool AI much if at all, and that it would be worthwhile to think about (e.g. because it already routinely exists), which Eliezer didn't really answer.

Comment author: JGWeissman 12 June 2012 05:01:44PM 2 points [-]

However, there's a weaker claim, that SIAI is not thinking about Tool AI much if at all, and that it would be worthwhile to think about (e.g. because it already routinely exists), which Eliezer didn't really answer.

Answering that was the point of section 3. Summary: Lots of other people also have their own favored solutions they think are obvious, none of which are also Tool AI. You shouldn't really expect that SIAI would have addressed your particular idea before you or anyone else even talked about it.

Comment author: private_messaging 13 June 2012 03:53:29AM *  -1 points [-]

I think the core distinction was poorly worded by Holden. The distinction is between AIs as they exist now (e.g. self driving car), and the economical model of AI within a larger model, as economical utility maximizer agent, a non-reductionistically modelled entity within a larger model, which is maximizing some utility non-reductionistically modelled within larger model (e.g. paperclip maximizer).

The AIs as they exist now, at the core, throw the 'intelligence' in form of solution search, at a problem of finding inputs to an internally defined mathematical function that produce the largest output value. Those inputs can be representing real world manipulator states, and output of the function can be representing the future metric of performance, but very loosely so. The intelligence is not thrown at the job of forming the best model of real world for making real world paperclips; the notion is not even coherent because the 'number of paperclips' is ill defined outside context of specific model of the world.

Comment author: Vaniver 12 June 2012 04:42:33AM -2 points [-]

Your link to Holden's post is broken.

It might be the right suggestion, but it's not so obviously right that our failure to prioritize discussing it reflects horrible negligence.

In a paragraph begging for charity, this sentence seems out of place.

(Commentary to follow.)

Comment author: ciphergoth 12 June 2012 06:44:55AM 6 points [-]

I can't see what you're getting at. Holden seems to say not just "you should do this", but "the fact that you're not already doing this reflects badly on your decision making". Eliezer replies that the first may be true but the second seems unwarranted.

Comment author: Vaniver 12 June 2012 03:56:00PM *  1 point [-]

Consider three sections of Holden's post:

Below, I list my major objections. I do not believe that these objections constitute a sharp/tight case for the idea that SI's work has low/negative value; I believe, instead, that SI's own arguments are too vague for such a rebuttal to be possible.

In section 1 and 2, Holden makes the argument that pinning our hopes on a utility function seems dangerous, because maximizers in general are dangerous. Better to just make information processing tools that make us more intelligent.

When discussing SI as an organization, Holden says,

One of SI's major goals is to raise awareness of AI-related risks; given this, the fact that it has not advanced clear/concise/compelling arguments speaks, in my view, to its general competence.

The jump from "speaks to its general competence" to "horribl[y] negligent" is a large and uncharitable one. If one focuses on "compelling," then yes, Holden is saying "SI is incompetent because I wasn't convinced by them," and that does seem unwarranted, or at least weak. But if one focuses on "clear" or "concise," then I agree with Holden- if SI's core mission is to communicate about AI risks, and they're unable to communicate clearly and concisely, then that speaks to their ability to complete their core mission! And there's the other bit where charity seemed lacking to me- it seems that Holden's strongest complaints are about clarity and concision.

Now, that's my impression as a bystander, and I "remember with compassion that it's not always obvious to one person what another person will think was the central point", so it is an observation about tone and little more.

Comment author: PhilGoetz 25 June 2012 08:40:47PM *  -3 points [-]

I don't think Eliezer addressed Holden's point about tool AI. My interpretation of Holden's point was, "SIAI should spend some time investigating the concept of Tool AI, see what can be done in that area to make something that is useful and safer than agentive AI, and promote the idea that AI should be pursued in that manner."

My interpretation of Eliezer's response (between <angle brackets>, because he won't like it if I use quotes) is,

A. <Other AI researchers have not found tool AI to be an obvious approach.>

This is completely irrelevant.

EDIT July 2: Eliezer's response would make sense only if Holden had been suggesting that SIAI should warn AI researchers against tool AI, as it warns them against autonomous AI. That was not what he was saying. He was saying that SIAI should consider tool AI as a possible more-safe kind of AI, just as it considers FAI as a possible more-safe kind of AI. If one rejects investigating tool AI because not many AI researchers use it, one must also reject investigating FAI, for the same reason.

ORIGINAL TEXT: Far many more AI researchers have found tool AI to be an obvious approach (e.g., Winograd, Schank, Lenat) than have found FAI to be an obvious approach. SIAI finds it worthwhile to investigate FAI, find some reasonable approach using it, and encourage other researchers to consider adopting that approach. Holden suggested that, in exactly the same way, SIAI could investigate tool AI, find some reasonable way of using it, and encourage other researchers to do that.

B. <My conception of tool AI, which would be vastly more powerful than any other conception of tool AI that anyone has considered, would secretly be an agentive AI because the difference between trying to inform the user and trying to manipulate the user is only semantic.>

Holden said, <Here's this vast space of possible kinds of AIs, subsumed under the term 'tool AI', that you should investigate.> And Eliezer replies, <AIs within a small subset of that space would be dangerous; therefore I'm not interested in that space.> There are lots of very-useful tool AIs that would not model the user. Google Maps takes two endpoints and produces a route between them. It relies on the user to have picked two endpoints that are useful. If Eliezer's objection were valid, he should have been able to come up with a scenario in which the algorithm Google Maps used to choose a route could pose a threat. It doesn't matter what else he says, if he can't show that.

Comment author: JGWeissman 25 June 2012 09:02:00PM 4 points [-]

A. <Other AI researchers have not found tool AI to be an obvious approach.>

That is not an argument against investigating Holden's idea. It is an explanation of why SIAI had not investigated Holden's idea before Holden had presented it (you can tell because it's in the section titled "Why we haven't already discussed Holden's suggestion"). This explanation was given in response Holden presenting the idea in the course of criticizing SIAI for not having investigated it.

Comment author: thomblake 25 June 2012 08:58:16PM 4 points [-]

If Eliezer's objection were valid, he should have been able to come up with a scenario in which the algorithm Google Maps used to choose a route could pose a threat. It doesn't matter what else he says, if he can't show that.

The discussion was about AGI. The algorithm the real Google Maps actually uses is irrelevant, since it is not an AGI. "Tool AI" does not simply mean "Narrow AI".

Comment author: PhilGoetz 26 June 2012 04:11:18AM *  -2 points [-]

The point is not what algorithm google maps uses. The point is that google maps does not model the user, and try to manipulate the user. Google maps is asked for a short way to get between two points, and it finds such a route and reports it. It is invulnerable to all the objections Eliezer makes, even though it is the example Eliezer began with when making his objections!

Comment author: TimS 25 June 2012 08:44:23PM *  3 points [-]

[EY] My particular conception of an extraordinarily powerful tool AI, which would be vastly more powerful than any other conception of tool AI that anyone has considered, would secretly be an agentive AI because the difference between trying to inform the user and trying to manipulate the user is only semantic.

This is not a valid response. Holden is saying, "Here's this vast space of possible kinds of AIs, subsumed under the term 'tool AI', that you should investigate." And Eliezer is saying, "AIs within a small subset of that space would be dangerous; therefore I'm not interested in that space."

How do you know it is a small subset? Or a subset at all? If every interestingly powerful tool AI is secretly an agent AI, that's bad, right?

Comment author: Dr_Manhattan 12 June 2012 03:15:18PM 1 point [-]

Demis Hassabis (VC-funded to the tune of several million dollars)

No public reference to his start-up that I can find.

Comment author: lukeprog 12 June 2012 03:29:41PM *  21 points [-]

They're still underground, with Shane Legg and at least a dozen other people on board. The company is called "Deep Mind" these days, and it's being developed as a games company. It's one of the most significant AGI projects I know of, merely because Shane and Demis are highly competent and approaching AGI by one of the more tractable paths (e.g. not AIXI or Goedel machines). Shane predicts AGI in a mere ten years—in part, I suspect, because he plans to build it himself.

Acquiring such facts is another thing SI does.

Comment author: Eliezer_Yudkowsky 12 June 2012 06:11:46PM 8 points [-]

I wouldn't endorse their significance the same way, and would stand by my statement that although the AGI field as a whole has perceptible risk, no individual project that I know of has perceptible risk. Shane and Demis are cool, but they ain't that cool.

Comment author: lukeprog 12 June 2012 06:21:11PM 8 points [-]

Right. I should have clarified that by "one of the most significant AGI projects I know of" I meant "has a very tiny probability of FOOMing in the next 15 years, which is greater than the totally negligible probability of FOOMing in the next 15 years posed by Juergen Schmidhuber."

Comment author: cousin_it 13 June 2012 09:54:14AM 2 points [-]

Googling for "hassabis legg deepmind" seems to reveal that Jaan Tallinn is also one of the directors there.

Comment author: MarkusRamikin 12 June 2012 06:31:25AM *  1 point [-]

The link to How to Purchase AI Risk Reduction, in part 4, seems to be not working.

EDIT: looks fixed now!

Comment author: lukeprog 22 March 2013 11:34:05PM 0 points [-]

Marcus Hutter is a rare exception who specified his AGI in such unambiguous mathematical terms that he actually succeeded at realizing, after some discussion with SIAI personnel, that AIXI would kill off its users and seize control of its reward button.

Not sure that's true of Hutter's beliefs, but for historical reference I'll link to a 2003 mailing list post by Eliezer describing some harmful consequences of AIXI-tl. Hutter wasn't part of that discussion, though.

Comment author: jacoblyles 18 July 2012 07:38:36AM 0 points [-]

What makes us think that AI would stick with the utility function they're given? I change my utility function all the time, sometimes on purpose.

Comment author: wedrifid 18 July 2012 10:42:36AM *  3 points [-]

What makes us think that AI would stick with the utility function they're given?

There are very few situations in which an agent can most effectively maximise expected utility according to their current utility function by modifying themselves to have a different utility function. Unless the AI is defective or put in a specially contrived scenario it will maintain its current utility function because that is an instrumentally useful thing to do.

If you are a paperclip maximiser then becoming a staples maximiser is a terribly inefficient strategy for maximising paperclips unless Omega is around making weird bargains.

I change my utility function all the time, sometimes on purpose.

No you don't. That is, to the extent that you "change your utility function" at all you do not have a utility function in sense meant when discussing AI. It only makes sense to model humans as having 'utility functions' when they are behaving in a manner that can be vaguely approximated as expected utility maximisers with a particular preference function.

Sure, it is possible to implement AIs that aren't expected utility maximisers either and those AIs could be made to do all sorts of arbitrary things including fundamentally change their goals and behavioral strategies. But if you implement an AI that tries to maximise a utility function then it will (almost always) keep trying to maximise that same utility function.

Comment author: ChristianKl 17 June 2012 09:30:06PM 0 points [-]

Inside the AI, whether an agent AI or a planning Oracle, there would be similar AGI-challenges like "build a predictive model of the world", and similar FAI-conjugates of those challenges like finding the 'user' inside an AI-created model of the universe.

Isn't building a predictive model of the world central to any AGI development? I don't see why someone who focuses specifically on FAI would worry more about a predictive model that other AGI developers. Specifically I don't think that even without Singularity Institute there would still be AGI people working on building predictive models of the world.

Comment author: timtyler 14 June 2012 12:47:17AM 0 points [-]

My mind keeps turning up Ben Goertzel as the one who invented this caricature - "Don't you understand, poor fool Eliezer, life is full of uncertainty, your attempt to flee from it by refuge in 'mathematical proof' is doomed" - but I'm not sure he was actually the inventor.

Of course, that is not a genuine quotation from Ben.