Press release

Computers are great at treating words as data: Word-processing programs let you rearrange and format text however you like, and search engines can quickly find a word anywhere on the Web. But what would it mean for a computer to actually understand the meaning of a sentence written in ordinary English — or French, or Urdu, or Mandarin?

One test might be whether the computer could analyze and follow a set of instructions for an unfamiliar task. And indeed, in the last few years, researchers at MIT’s Computer Science and Artificial Intelligence Lab have begun designing machine-learning systems that do exactly that, with surprisingly good results.

In 2009, at the annual meeting of the Association for Computational Linguistics (ACL), researchers in the lab of Regina Barzilay, associate professor of computer science and electrical engineering, took the best-paper award for a system that generated scripts for installing a piece of software on a Windows computer by reviewing instructions posted on Microsoft’s help site. At this year’s ACL meeting, Barzilay, her graduate student S. R. K. Branavan and David Silver of University College London applied a similar approach to a more complicated problem: learning to play “Civilization,” a computer game in which the player guides the development of a city into an empire across centuries of human history. When the researchers augmented a machine-learning system so that it could use a player’s manual to guide the development of a game-playing strategy, its rate of victory jumped from 46 percent to 79 percent.

Starting from scratch

“Games are used as a test bed for artificial-intelligence techniques simply because of their complexity,” says Branavan, who was first author on both ACL papers. “Every action that you take in the game doesn’t have a predetermined outcome, because the game or the opponent can randomly react to what you do. So you need a technique that can handle very complex scenarios that react in potentially random ways.”

Moreover, Barzilay says, game manuals have “very open text. They don’t tell you how to win. They just give you very general advice and suggestions, and you have to figure out a lot of other things on your own.” Relative to an application like the software-installing program, Branavan explains, games are “another step closer to the real world.”

The extraordinary thing about Barzilay and Branavan’s system is that it begins with virtually no prior knowledge about the task it’s intended to perform or the language in which the instructions are written. It has a list of actions it can take, like right-clicks or left-clicks, or moving the cursor; it has access to the information displayed on-screen; and it has some way of gauging its success, like whether the software has been installed or whether it wins the game. But it doesn’t know what actions correspond to what words in the instruction set, and it doesn’t know what the objects in the game world represent.

So initially, its behavior is almost totally random. But as it takes various actions, different words appear on screen, and it can look for instances of those words in the instruction set. It can also search the surrounding text for associated words, and develop hypotheses about what actions those words correspond to. Hypotheses that consistently lead to good results are given greater credence, while those that consistently lead to bad results are discarded.

Proof of concept

In the case of software installation, the system was able to reproduce 80 percent of the steps that a human reading the same instructions would execute. In the case of the computer game, it won 79 percent of the games it played, while a version that didn't rely on the written instructions won only 46 percent. The researchers also tested a more-sophisticated machine-learning algorithm that eschewed textual input but used additional techniques to improve its performance. Even that algorithm won only 62 percent of its games.

“If you’d asked me beforehand if I thought we could do this yet, I’d have said no,” says Eugene Charniak, University Professor of Computer Science at Brown University. “You are building something where you have very little information about the domain, but you get clues from the domain itself.”

Charniak points out that when the MIT researchers presented their work at the ACL meeting, some members of the audience argued that more sophisticated machine-learning systems would have performed better than the ones to which the researchers compared their system. But, Charniak adds, “it’s not completely clear to me that that’s really relevant. Who cares? The important point is that this was able to extract useful information from the manual, and that’s what we care about.”

Most computer games as complex as “Civilization” include algorithms that allow players to play against the computer, rather than against other people; the games’ programmers have to develop the strategies for the computer to follow and write the code that executes them. Barzilay and Branavan say that, in the near term, their system could make that job much easier, automatically creating algorithms that perform better than the hand-designed ones.

But the main purpose of the project, which was supported by the National Science Foundation, was to demonstrate that computer systems that learn the meanings of words through exploratory interaction with their environments are a promising subject for further research. And indeed, Barzilay and her students have begun to adapt their meaning-inferring algorithms to work with robotic systems.

The actual paper

Abstract

This paper presents a novel approach for leveraging automatically extracted textual knowledge to improve the performance of control applications such as games. Our ultimate goal is to enrich a stochastic player with highlevel guidance expressed in text. Our model jointly learns to identify text that is relevant to a given game state in addition to learning game strategies guided by the selected text. Our method operates in the Monte-Carlo search framework, and learns both text analysis and game strategies based only on environment feedback. We apply our approach to the complex strategy game Civilization II using the official game manual as the text guide. Our results show that a linguistically-informed game-playing agent significantly outperforms its language-unaware counterpart, yielding a 27% absolute improvement and winning over 78% of games when playing against the builtin AI of Civilization II.

The team has been rather exemplary when it comes to scientific openness: "The code, data and complete experimental setup for this work are available at http://groups.csail.mit.edu/rbg/code/civ ." And although I haven't downloaded it because it's a 1,1 gigabyte file, the page appears to contain a virtual machine with everything you need to actually run the experiment. I was expecting to a see a bunch of different, poorly documented files that you would have a hell of a time running if you didn't have exactly the same system as the researchers, so this was quite a pleasant surprise. (Of course, if I downloaded the VM it could be that it's still exactly that bad. :)

Restoring Memory, Repairing Damaged Brains (article @ PR Newswire)

Using an electronic system that duplicates the neural signals associated with memory, they managed to replicate the brain function in rats associated with long-term learned behavior, even when the rats had been drugged to forget.

This series of experiments, as described, sounds very well-constructed and thorough. The scientists first recorded specific activity in the hippocampus, where short-term memory becomes long-term memory. They then used drugs to inhibit that activity, preventing the formation of and access to long-term memory. Using the information they had gathered about the hippocampus activity, they constructed an artificial replacement and implanted it into the rats' brains. This successfully restored the rats' ability to store and use long-term memory. Further, they implanted the device into rats without suppressed hippocampal activity, and demonstrated increased memory abilities in those subjects.

"These integrated experimental modeling studies show for the first time that with sufficient information about the neural coding of memories, a neural prosthesis capable of real-time identification and manipulation of the encoding process can restore and even enhance cognitive mnemonic processes," says the paper.

It's a truly impressive result.

[Click here to see a list of all interviews]

I am emailing experts in order to raise and estimate the academic awareness and perception of risks from AI.

Below you will find some thoughts on the topic by Jürgen Schmidhuber, a computer scientist and AI researcher who wants to build an optimal scientist and then retire.

The Interview:

Q: What probability do you assign to the possibility of us being wiped out by badly done AI?

Jürgen Schmidhuber: Low for the next few months.

Q: What probability do you assign to the possibility of a human level AI, respectively sub-human level AI, to self-modify its way up to massive superhuman intelligence within a matter of hours or days?

Jürgen Schmidhuber: High for the next few decades, mostly because some of our own work seems to be almost there:

• Gödel machine: http://www.idsia.ch/~juergen/goedelmachine.html
• Universal AI: http://www.idsia.ch/~juergen/unilearn.html
• Creative machines that create and solve their own problems [4,5] to improve their knowledge about how the world works: http://www.idsia.ch/~juergen/creativity.html

Q: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?

Jürgen Schmidhuber: From a paper of mine:

All attempts at making sure there will be only provably friendly AIs seem doomed. Once somebody posts the recipe for practically feasible self-improving Goedel machines or AIs in form of code into which one can plug arbitrary utility functions, many users will equip such AIs with many different goals, often at least partially conflicting with those of humans. The laws of physics and the availability of physical resources will eventually determine which utility functions will help their AIs more than others to multiply and become dominant in competition with AIs driven by different utility functions. Which values are "good"? The survivors will define this in hindsight, since only survivors promote their values.

Q: What is the current level of awareness of possible risks from AI within the artificial intelligence community, relative to the ideal level?

Jürgen Schmidhuber: Some are interested in this, but most don't think it's relevant right now.

Q: How do risks from AI compare to other existential risks, e.g. advanced nanotechnology?

Jürgen Schmidhuber: I guess AI risks are less predictable.

(In his response to my questions he also added the following.)

Jürgen Schmidhuber: Recursive Self-Improvement: The provably optimal way of doing this was published in 2003. From a recent survey paper:

The fully self-referential Goedel machine [1,2] already is a universal AI that is at least theoretically optimal in a certain sense. It may interact with some initially unknown, partially observable environment to maximize future expected utility or reward by solving arbitrary user-defined computational tasks. Its initial algorithm is not hardwired; it can completely rewrite itself without essential limits apart from the limits of computability, provided a proof searcher embedded within the initial algorithm can first prove that the rewrite is useful, according to the formalized utility function taking into account the limited computational resources. Self-rewrites may modify / improve the proof searcher itself, and can be shown to be globally optimal, relative to Goedel's well-known fundamental restrictions of provability. To make sure the Goedel machine is at least asymptotically optimal even before the first self-rewrite, we may initialize it by Hutter's non-self-referential but asymptotically fastest algorithm for all well-defined problems HSEARCH [3], which uses a hardwired brute force proof searcher and (justifiably) ignores the costs of proof search. Assuming discrete input/output domains X/Y, a formal problem specification f : X -> Y (say, a functional description of how integers are decomposed into their prime factors), and a particular x in X (say, an integer to be factorized), HSEARCH orders all proofs of an appropriate axiomatic system by size to find programs q that for all z in X provably compute f(z) within time bound tq(z). Simultaneously it spends most of its time on executing the q with the best currently proven time bound tq(x). Remarkably, HSEARCH is as fast as the fastest algorithm that provably computes f(z) for all z in X, save for a constant factor smaller than 1 + epsilon (arbitrary real-valued epsilon > 0) and an f-specific but x-independent additive constant. Given some problem, the Goedel machine may decide to replace its HSEARCH initialization by a faster method suffering less from large constant overhead, but even if it doesn't, its performance won't be less than asymptotically optimal.

All of this implies that there already exists the blueprint of a Universal AI which will solve almost all problems almost as quickly as if it already knew the best (unknown) algorithm for solving them, because almost all imaginable problems are big enough to make the additive constant negligible. The only motivation for not quitting computer science research right now is that many real-world problems are so small and simple that the ominous constant slowdown (potentially relevant at least before the first Goedel machine self-rewrite) is not negligible. Nevertheless, the ongoing efforts at scaling universal AIs down to the rather few small problems are very much informed by the new millennium's theoretical insights mentioned above, and may soon yield practically feasible yet still general problem solvers for physical systems with highly restricted computational power, say, a few trillion instructions per second, roughly comparable to a human brain power.

[1] J. Schmidhuber. Goedel machines: Fully Self-Referential Optimal Universal Self-Improvers. In B. Goertzel and C. Pennachin, eds.: Artificial General Intelligence, p. 119-226, 2006.

[2] J. Schmidhuber. Ultimate cognition à la Goedel. Cognitive Computation, 1(2):177-193, 2009.

[3] M. Hutter. The fastest and shortest algorithm for all well-defined problems. International Journal of
Foundations of Computer Science, 13(3):431-443, 2002. (On J. Schmidhuber's SNF grant 20-61847).

[4] J. Schmidhuber. Developmental robotics, optimal artificial curiosity, creativity, music, and the fine
arts. Connection Science, 18(2):173-187, 2006.

[5] J. Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions
on Autonomous Mental Development, 2(3):230-247, 2010.

A dozen earlier papers on (not yet theoretically optimal) recursive self-improvement since 1987 are here: http://www.idsia.ch/~juergen/metalearner.html

Anonymous

At this point I would also like to give a short roundup. Most experts I wrote haven't responded at all so far, although a few did but asked me not to publish their answers. Some of them are well-known even outside of their field of expertise and respected even here on LW.

I will paraphrase some of the responses I got below:

Anonymous expert 01: I think the so-called Singularity is unlikely to come about in the foreseeable future. I already know about the SIAI and I think that the people who are involved with it are well-meaning, thoughtful and highly intelligent. But I personally think that they are naïve as far as the nature of human intelligence goes. None of them seems to have a realistic picture about the nature of thinking.

Anonymous expert 02: My opinion is that some people hold much stronger opinions on this issue than justified by our current state of knowledge.

Anonymous expert 03: I believe that the biggest risk from AI is that at some point we will become so dependent on it that we lose our cognitive abilities. Today people are losing their ability to navigate with maps, thanks to GPS. But such a loss will be nothing compared to what we might lose by letting AI solve more important problems for us.

Anonymous expert 04:  I think these are nontrivial questions and that risks from AI have to be taken seriously. But I also believe that many people have made scary-sounding but mostly unfounded speculations. In principle an AI could take over the world, but currently AI presents no threat. At some point, it will become a more pressing issue. In the mean time, we are much more likely to destroy ourselves by other means.

There are a lot of bad arguments out there. Fortunately, there are also plenty of people who stand up against these arguments, which is good.

However, there is a pattern I observe quite often in such counter-arguments, which, while strictly logically valid, can become problematic later. It involves fixing all of one's counter-arguments on countering one, and only one, of the original arguer's points. I suspect that this tendency can, at best, weaken one's argument, and, at worst, allow oneself to believe things one has no intention of believing.

Let's assume, without much loss of generality, that the Wrong Argument can be expressed in the following form:

A: Some statement.
B: Some other statement.
A & B -> C: A logical inference, which, from the way B is constructed, is a fairly obvious tautology.
C: The conclusion.

Unfortunately, most of the arguments I could choose for this discussion are either highly trivial or highly controversial. I'll choose one that I hope won't cause too much trouble. Bear in mind that this is the Wrong Argument, the thing that the counter-arguer, the person presenting the good, rational refutation, is trying to demonstrate to be false. Let's designate this rational arguer as RA. The person presenting the Wrong Argument will be designated WA (Wrong Arguer).

WA: "Men have better technical abilities than women, so they should get paid more for the same engineering jobs."

WA relates a terrible sentiment, yet a pervasive one. I don't know anyone who actually espouses it in my workplace, but it was certainly commonplace not so long ago (musical evidence). Let's hope that RA has something persuasive to say against it.

Based on what I've seen of gender discussions on other forums, here's the most likely response I'd expect from RA:

RA: Don't be ridiculous! Men and women are just as well suited to technical careers as each other!

... and that's usually as far as it goes. Now, RA is right, as far as anyone knows (IANAPsychologist, though).

However, WA's argument can be broken down into the following steps:

A: Men, on average, have better technical skills than women.
B: If members of one group, on average, are better at a task than members of another group, then members of that first group should be paid more than members of the second group for performing the same work.
C: Men should be paid more than women for the same work in technical fields such as engineering.

Trivially, A & B -> C. Thus RA only needs to disprove A or B in order to break the argument. (Yes, ~A doesn't imply ~C, but WA will have a hard time proving C without A.) Both A and B are unpleasant statements that decent, rational people should probably disagree with, and C is definitely problematic.

So RA sets about attacking A. He starts by simply stating that men and women have equal potential for technical talent, on average. If WA doesn't believe that, then RA presents anecdotal evidence, then starts digging up psychological studies. Every rational discourse weapon at RA's disposal may be deployed to show that A is false. Maybe WA will be convinced, maybe he won't.

But what about B? RA has ignored B entirely in his attack on A. Now, from a strictly logical point of view, RA doesn't need to do anything with B - if he disproves A, then he disproves A & B. Attacking A doesn't mean that he accepts B as true...

... except that it kind of does.

What if WA manages to win the argument over A, by whatever means? What if WA turns out to be an evolutionary psychology clever arguer, with several papers worth of "evidence" that "proves" that men have better technical skills than women? RA might simply not have the skills or resources to refute WA's points, leading to the following exchange:

WA: Men are better engineers than women, and should be paid more!

RA: That's ridiculous. Men and women have identical potentials for technical skill!

WA: No they don't! Here are ten volumes' worth of papers proving me right!

RA: Well, gee, who am I to argue with psychology journals? I guess you're right.

WA: Glad we agree. I'll go talk to the CTO about Wanda's pay cut, shall I?

RA: Hang on a minute! Even if men are better engineers than women, that's no reason for pay inequity! Equal work for equal pay is the only fair way. If men really are better, they'll get raises and promotions on their own merit, not merely by virtue of being male.

WA: What? I spent hours getting those references together, and now you've moved the goalposts on me! I thought you weren't meant to do that!

RA: But... it's true...

WA: I think you've just taken your conclusion, "Men and women should get equal pay for the same work", and figured out a line of reasoning that gets you there. What are you, some kind of clever arguer for female engineers? Wait, isn't your mother an engineer too?

Nobody wants to be in this situation. RA really has moved the goalposts on WA, which is one of those Dark Arts that we're not supposed to employ, even unintentionally.

The problem goes deeper than simply violating good debating etiquette, though. If this debate is happening in public, then onlookers might get the impression that RA supports B. It will then be more difficult for RA to argue against B in later arguments, especially ones of the form D & B, where D is actually true. (For example, D might be "Old engineers have better technical skills than younger engineers", which is true-ish because of the benefits of long experience in an industry, but it still shouldn't mean that old engineers automatically deserve higher pay for the same work.)

Furthermore, and again IANAP, but it seems possible to me that if RA keeps arguing against A and ignoring B, he might actually start believing B. Alternatively, he might not specifically believe B, but he might stop thinking about B at all, and start ignoring the B step in his own reasoning and other people's.

So, the way to avoid all of this, is to raise all of your objections simultaneously, thusly:

WA: Men are better engineers than women, and should be paid more!

RA: Woah. Okay, first? There's no evidence to suggest that that's actually true. But secondly, even pretending for the moment that it were true, that would be no excuse for paying women less for the same work.

WA: Oh. Um. I'm pretty confident about that first point, but I never actually thought I'd have to defend the other bit. I'll go away now.

That's a best-case scenario, but it does avoid the problems above.

This post has already turned out longer than I intended, so I'll end it here. The last point I wanted to raise, though, is that an awful lot of Wrong Arguments (or good arguments, for that matter) take a form where A is an assertion of fact ("men are better engineers than women"), and B is an expression of morality ("... and therefore they should get paid more"). There are some important implications to this, for which I have a number of examples to present if people are interested.

To summarise: If someone says "A and B are true!", don't just say "A isn't true!". Say "A isn't true, and even if it were, B isn't true either!". Otherwise people might think you believe B, and they might even be right.

Related to: People who want to save the world 

I have recently been diagnosed with cancer, for which I am currently being treated with good prognosis. I've been reevaluating my life plans and priorities in response. To be clear, I estimate that the cancer is responsible for much less than half the total danger to my life. The universals - X-risks, diseases I don't have yet, traffic accidents, etc. - are worse.

I would like to affirm my desire to Save Myself (and Save The World For Myself). Saving the world is a prerequisite simply because the world is in danger. I believe my values are well aligned with those of the LW community; wanting to Save The World is a good applause light but I believe most people want to do so for selfish reasons. 

I would also like to ask LW members: why do you prefer to contribute (in part) towards humankind-wide X-risk problems rather than more narrow but personally important issues? How do you determine the time- and risk- tradeoffs between things like saving money for healthcare, and investing money in preventing an unfriendly AI FOOM?

It is common advice here to focus on earning money and donating it to research, rather than donating in kind. How do you decide what portion of income to donate to SIAI, which to SENS, and which to keep as money for purely personal problems that others won't invest in? There's no conceptual difficulty here, but I have no idea how to quantify the risks involved.

I recently recalled, apropos of the intermittent fasting/caloric restriction discussion, a very good blog post on mortality curves and models of aging:

For me, a 25-year-old American, the probability of dying during the next year is a fairly miniscule 0.03% — about 1 in 3,000.  When I’m 33 it will be about 1 in 1,500, when I’m 42 it will be about 1 in 750, and so on.  By the time I reach age 100 (and I do plan on it) the probability of living to 101 will only be about 50%.  This is seriously fast growth — my mortality rate is increasing exponentially with age.

...This data fits the Gompertz law almost perfectly, with death rates doubling every 8 years.  The graph on the right also agrees with the Gompertz law, and you can see the precipitous fall in survival rates starting at age 80 or so.  That decline is no joke; the sharp fall in survival rates can be expressed mathematically as an exponential within an exponential:

Exponential decay is sharp, but an exponential within an exponential is so sharp that I can say with 99.999999% certainty that no human will ever live to the age of 130.  (Ignoring, of course, the upward shift in the lifetime distribution that will result from future medical advances)

...There is one important lesson, however, to be learned from Benjamin Gompertz’s mysterious observation.  By looking at theories of human mortality that are clearly wrong, we can deduce that our fast-rising mortality is not the result of a dangerous environment, but of a body that has a built-in expiration date.

gravityandlevity then discusses some simple models of aging and the statistical characters they have which do not match Gompertz's law:

1. 'lightning' model: risk of mortality each period is constant; Poisson distribution:
   

   What a crazy world!  The average lifespan would be the same, but out of every 100 people 31 would die before age 30 and 2 of them would live to be more than 300 years old.  Clearly we do not live in a world where mortality is governed by “lightning bolts”.

2. 'accumulated lightning'; like in a video game, one has a healthbar which may take a hit each period; similar to above:
   

   Shown above are the results from a simulated world where “lightning bolts” of misfortune hit people on average every 16 years, and death occurs at the fifth hit.  This world also has an average lifespan of 80 years (16*5 = 80), and its distribution is a little less ridiculous than the previous case.  Still, it’s no Gompertz Law: look at all those 160-year-olds!  You can try playing around with different “lightning strike rates” and different number of hits required for death, but nothing will reproduce the Gompertz Law.  No explanation based on careless gods, no matter how plentiful or how strong their blows are, will reproduce the strong upper limit to human lifespan that we actually observe.

What models do yield a Gompertz curve? gravityandlevity describes a simple 'cops and robbers' model (which I like to think of as 'antibodies and cancers'):

...in general, the cops are winning.  They patrol randomly through your body, and when they happen to come across a criminal he is promptly removed.  The cops can always defeat a criminal they come across, unless the criminal has been allowed to sit in the same spot for a long time.  A criminal that remains in one place for long enough (say, one day) can build a “fortress” which is too strong to be assailed by the police.  If this happens, you die.

Lucky for you, the cops are plentiful, and on average they pass by every spot 14 times a day.  The likelihood of them missing a particular spot for an entire day is given (as you’ve learned by now) by the Poisson distribution: it is a mere .

But what happens if your internal police force starts to dwindle?  Suppose that as you age the police force suffers a slight reduction, so that they can only cover every spot 12 times a day.  Then the probability of them missing a criminal for an entire day decreases to .  The difference between 14 and 12 doesn’t seem like a big deal, but the result was that your chance of dying during a given day jumped by more than 10 times.  And if the strength of your police force drops linearly in time, your mortality rate will rise exponentially.

... The language of “cops and criminals” lends itself very easily to a discussion of the immune system fighting infection and random mutation.  Particularly heartening is the fact that rates of cancer incidence also follow the Gompertz law, doubling every 8 years or so.  Maybe something in the immune system is degrading over time, becoming worse at finding and destroying mutated and potentially dangerous cells.

...Who are the criminals and who are the cops that kill them?  What is the “incubation time” for a criminal, and why does it give “him” enough strength to fight off the immune response?  Why is the police force dwindling over time?  For that matter, what kind of “clock” does your body have that measures time at all? There have been attempts to describe DNA degradation (through the shortening of your telomeres or through methylation) as an increase in “criminals” that slowly overwhelm the body’s DNA-repair mechanisms, but nothing has come of it so far.

This offers food for thought about various anti-aging strategies. For example, given the superexponential growth in mortality, if we had a magic medical treatment that could cut your mortality risk in half but didn't affect the growth of said risk, then that would buy you very little late in life, but might extend life by decades if administered at a very young age.

I've written before about the difficulty of distinguishing values from errors, from algorithms, and from context.  Now I have to add to that list:  How can we distinguish our utility function from the parameters we use to apply it?

I've often wondered about people who appear to be very smart, and do very stupid things.  One theory is that people are smart and stupid independently in different domains.  Another theory is that "smart" and "stupid" are oversimplifications.  In line with the second theory, here is an ad-hoc set of axes of intelligence, based only on my own observations.