Superintelligence 7: Decisive strategic advantage

KatjaGrace

24 Superintelligence 7: Decisive strategic advantage

28th Oct 2014

7 min read

24

This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.

Welcome. This week we discuss the seventh section in the reading guide: Decisive strategic advantage. This corresponds to Chapter 5.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.

There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).

Reading: Chapter 5 (p78-91)

Summary

Question: will a single artificial intelligence project get to 'dictate the future'? (p78)
We can ask, will a project attain a 'decisive strategic advantage' and will they use this to make a 'singleton'?
1. 'Decisive strategic advantage' = a level of technological and other advantages sufficient for complete world domination (p78)
2. 'Singleton' = a single global decision-making agency strong enough to solve all major global coordination problems (p78, 83)
A project will get a decisive strategic advantage if there is a big enough gap between its capability and that of other projects.
A faster takeoff would make this gap bigger. Other factors would too, e.g. diffusion of ideas, regulation or expropriation of winnings, the ease of staying ahead once you are far enough ahead, and AI solutions to loyalty issues (p78-9)
For some historical examples, leading projects have a gap of a few months to a few years with those following them. (p79)
Even if a second project starts taking off before the first is done, the first may emerge decisively advantageous. If we imagine takeoff accelerating, a project that starts out just behind the leading project might still be far inferior when the leading project reaches superintelligence. (p82)
How large would a successful project be? (p83) If the route to superintelligence is not AI, the project probably needs to be big. If it is AI, size is less clear. If lots of insights are accumulated in open resources, and can be put together or finished by a small team, a successful AI project might be quite small (p83).
We should distinguish the size of the group working on the project, and the size of the group that controls the project (p83-4)
If large powers anticipate an intelligence explosion, they may want to monitor those involved and/or take control. (p84)
It might be easy to monitor very large projects, but hard to trace small projects designed to be secret from the outset. (p85)
Authorities may just not notice what's going on, for instance if politically motivated firms and academics fight against their research being seen as dangerous. (p85)
Various considerations suggest a superintelligence with a decisive strategic advantage would be more likely than a human group to use the advantage to form a singleton (p87-89)

Another view

This week, Paul Christiano contributes a guest sub-post on an alternative perspective:

Typically new technologies do not allow small groups to obtain a “decisive strategic advantage”—they usually diffuse throughout the whole world, or perhaps are limited to a single country or coalition during war. This is consistent with intuition: a small group with a technological advantage will still do further research slower than the rest of the world, unless their technological advantage overwhelms their smaller size.

The result is that small groups will be overtaken by big groups. Usually the small group will sell or lease their technology to society at large first, since a technology’s usefulness is proportional to the scale at which it can be deployed. In extreme cases such as war these gains might be offset by the cost of empowering the enemy. But even in this case we expect the dynamics of coalition-formation to increase the scale of technology-sharing until there are at most a handful of competing factions.

So any discussion of why AI will lead to a decisive strategic advantage must necessarily be a discussion of why AI is an unusual technology.

In the case of AI, the main difference Bostrom highlights is the possibility of an abrupt increase in productivity. In order for a small group to obtain such an advantage, their technological lead must correspond to a large productivity improvement. A team with a billion dollar budget would need to secure something like a 10,000-fold increase in productivity in order to outcompete the rest of the world. Such a jump is conceivable, but I consider it unlikely. There are other conceivable mechanisms distinctive to AI; I don’t think any of them have yet been explored in enough depth to be persuasive to a skeptical audience.

Notes

1. Extreme AI capability does not imply strategic advantage. An AI program could be very capable - such that the sum of all instances of that AI worldwide were far superior (in capability, e.g. economic value) to the rest of humanity's joint efforts - and yet the AI could fail to have a decisive strategic advantage, because it may not be a strategic unit. Instances of the AI may be controlled by different parties across society. In fact this is the usual outcome for technological developments.

2. On gaps between the best AI project and the second best AI project (p79) A large gap might develop either because of an abrupt jump in capability or extremely fast progress (which is much like an abrupt jump), or from one project having consistent faster growth than other projects for a time. Consistently faster progress is a bit like a jump, in that there is presumably some particular highly valuable thing that changed at the start of the fast progress. Robin Hanson frames his Foom debate with Eliezer as about whether there are 'architectural' innovations to be made, by which he means innovations which have a large effect (or so I understood from conversation). This seems like much the same question. On this, Robin says:

Yes, sometimes architectural choices have wider impacts. But I was an artificial intelligence researcher for nine years, ending twenty years ago, and I never saw an architecture choice make a huge difference, relative to other reasonable architecture choices. For most big systems, overall architecture matters a lot less than getting lots of detail right. Researchers have long wandered the space of architectures, mostly rediscovering variations on what others found before.

3. What should activists do? Bostrom points out that activists seeking maximum expected impact might wish to focus their planning on high leverage scenarios, where larger players are not paying attention (p86). This is true, but it's worth noting that changing the probability of large players paying attention is also an option for activists, if they think the 'high leverage scenarios' are likely to be much better or worse.

4. Trade. One key question seems to be whether successful projects are likely to sell their products, or hoard them in the hope of soon taking over the world. I doubt this will be a strategic decision they will make - rather it seems that one of these options will be obviously better given the situation, and we are uncertain about which. A lone inventor of writing should probably not have hoarded it for a solitary power grab, even though it could reasonably have seemed like a good candidate for radically speeding up the process of self-improvement.

5. Disagreement. Note that though few people believe that a single AI project will get to dictate the future, this is often because they disagree with things in the previous chapter - e.g. that a single AI project will plausibly become more capable than the world in the space of less than a month.

6. How big is the AI project? Bostrom distinguishes between the size of the effort to make AI and the size of the group ultimately controlling its decisions. Note that the people making decisions for the AI project may also not be the people making decisions for the AI - i.e. the agents that emerge. For instance, the AI making company might sell versions of their AI to a range of organizations, modified for their particular goals. While in some sense their AI has taken over the world, the actual agents are acting on behalf of much of society.

In-depth investigations

If you are particularly interested in these topics, and want to do further research, these are a few plausible directions, some inspired by Luke Muehlhauser's list, which contains many suggestions related to parts of Superintelligence. These projects could be attempted at various levels of depth.

When has anyone gained a 'decisive strategic advantage' at a smaller scale than the world? Can we learn anything interesting about what characteristics a project would need to have such an advantage with respect to the world?
How scalable is innovative project secrecy? Examine past cases: Manhattan project, Bletchly park, Bitcoin, Anonymous, Stuxnet, Skunk Works, Phantom Works, Google X.
How large are the gaps in development time between modern software projects? What dictates this? (e.g. is there diffusion of ideas from engineers talking to each other? From people changing organizations? Do people get far enough ahead that it is hard to follow them?)

If you are interested in anything like this, you might want to mention it in the comments, and see whether other people have useful thoughts.

How to proceed

This has been a collection of notes on the chapter. The most important part of the reading group though is discussion, which is in the comments section. I pose some questions for you there, and I invite you to add your own. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

Next week, we will talk about Cognitive superpowers (section 8). To prepare, read Chapter 6. The discussion will go live at 6pm Pacific time next Monday 3 November. Sign up to be notified here.

SuperintelligenceAI GovernanceReading GroupAI

Personal Blog

24

New Comment

Rendering 0/60 comments, sorted by

top scoring

(show more) Click to highlight new comments since: Today at 5:21 AM

Moderation Log

24 Superintelligence 7: Decisive strategic advantage

by KatjaGrace

28th Oct 2014

7 min read

24

Welcome. This week we discuss the seventh section in the reading guide: Decisive strategic advantage. This corresponds to Chapter 5.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.

Reading: Chapter 5 (p78-91)

Summary

Question: will a single artificial intelligence project get to 'dictate the future'? (p78)
We can ask, will a project attain a 'decisive strategic advantage' and will they use this to make a 'singleton'?
1. 'Decisive strategic advantage' = a level of technological and other advantages sufficient for complete world domination (p78)
2. 'Singleton' = a single global decision-making agency strong enough to solve all major global coordination problems (p78, 83)
A project will get a decisive strategic advantage if there is a big enough gap between its capability and that of other projects.
A faster takeoff would make this gap bigger. Other factors would too, e.g. diffusion of ideas, regulation or expropriation of winnings, the ease of staying ahead once you are far enough ahead, and AI solutions to loyalty issues (p78-9)
For some historical examples, leading projects have a gap of a few months to a few years with those following them. (p79)
Even if a second project starts taking off before the first is done, the first may emerge decisively advantageous. If we imagine takeoff accelerating, a project that starts out just behind the leading project might still be far inferior when the leading project reaches superintelligence. (p82)
How large would a successful project be? (p83) If the route to superintelligence is not AI, the project probably needs to be big. If it is AI, size is less clear. If lots of insights are accumulated in open resources, and can be put together or finished by a small team, a successful AI project might be quite small (p83).
We should distinguish the size of the group working on the project, and the size of the group that controls the project (p83-4)
If large powers anticipate an intelligence explosion, they may want to monitor those involved and/or take control. (p84)
It might be easy to monitor very large projects, but hard to trace small projects designed to be secret from the outset. (p85)
Authorities may just not notice what's going on, for instance if politically motivated firms and academics fight against their research being seen as dangerous. (p85)
Various considerations suggest a superintelligence with a decisive strategic advantage would be more likely than a human group to use the advantage to form a singleton (p87-89)

Another view

This week, Paul Christiano contributes a guest sub-post on an alternative perspective:

Typically new technologies do not allow small groups to obtain a “decisive strategic advantage”—they usually diffuse throughout the whole world, or perhaps are limited to a single country or coalition during war. This is consistent with intuition: a small group with a technological advantage will still do further research slower than the rest of the world, unless their technological advantage overwhelms their smaller size.

The result is that small groups will be overtaken by big groups. Usually the small group will sell or lease their technology to society at large first, since a technology’s usefulness is proportional to the scale at which it can be deployed. In extreme cases such as war these gains might be offset by the cost of empowering the enemy. But even in this case we expect the dynamics of coalition-formation to increase the scale of technology-sharing until there are at most a handful of competing factions.

So any discussion of why AI will lead to a decisive strategic advantage must necessarily be a discussion of why AI is an unusual technology.

In the case of AI, the main difference Bostrom highlights is the possibility of an abrupt increase in productivity. In order for a small group to obtain such an advantage, their technological lead must correspond to a large productivity improvement. A team with a billion dollar budget would need to secure something like a 10,000-fold increase in productivity in order to outcompete the rest of the world. Such a jump is conceivable, but I consider it unlikely. There are other conceivable mechanisms distinctive to AI; I don’t think any of them have yet been explored in enough depth to be persuasive to a skeptical audience.

Notes

Yes, sometimes architectural choices have wider impacts. But I was an artificial intelligence researcher for nine years, ending twenty years ago, and I never saw an architecture choice make a huge difference, relative to other reasonable architecture choices. For most big systems, overall architecture matters a lot less than getting lots of detail right. Researchers have long wandered the space of architectures, mostly rediscovering variations on what others found before.

In-depth investigations

When has anyone gained a 'decisive strategic advantage' at a smaller scale than the world? Can we learn anything interesting about what characteristics a project would need to have such an advantage with respect to the world?
How scalable is innovative project secrecy? Examine past cases: Manhattan project, Bletchly park, Bitcoin, Anonymous, Stuxnet, Skunk Works, Phantom Works, Google X.
How large are the gaps in development time between modern software projects? What dictates this? (e.g. is there diffusion of ideas from engineers talking to each other? From people changing organizations? Do people get far enough ahead that it is hard to follow them?)

If you are interested in anything like this, you might want to mention it in the comments, and see whether other people have useful thoughts.

How to proceed

Next week, we will talk about Cognitive superpowers (section 8). To prepare, read Chapter 6. The discussion will go live at 6pm Pacific time next Monday 3 November. Sign up to be notified here.

SuperintelligenceAI GovernanceReading GroupAI

Personal Blog

24

Mentioned in

222An overview of 11 proposals for building safe advanced AI

184Cortés, Pizarro, and Afonso as Precedents for Takeover

79AI strategy nearcasting

58Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios

56A "weak" AGI may attempt an unlikely-to-succeed takeover

Load More (5/9)

New Comment

Rendering 0/60 comments, sorted by

top scoring

(show more) Click to highlight new comments since: Today at 5:21 AM

Moderation Log

More from KatjaGrace

Curated and popular this week

60Comments

Comment Permalink

paulfchristiano12y10

Note that a 10,000-fold increase would leave you with a modest fraction of total output. A 1,000-fold increase would still leave you smaller than world's militaries.

The theft scenario relies on you being better at theft than the rest of the world. From the outside AI looks more suited to productive activity than war (which also requires big capital investments in personnel and equipment), so I normally think of this as being a counterbalancing factor (that is, it seems significantly more likely that an economically dominant firm would find many of its productive assets confiscated by force, than that a productive firm would begin stealing the world’s resources without serious repercussions). Of course a primary consideration in this discussion is the nature of conflict; in the current world a sufficiently sophisticated AI might fare reasonably well in all-out conflict due to the world being completely unprepared, but I would be quite surprised if that were still the case when the development of human-level AI actually looked plausible. It seems to again come down to the possibility of a rapid and unexpected jump in capabilities.

The most realistic paths along these lines seem to depend on abrupt changes in the importance of different resources. For example, you might think that AI research capacity would change from a small part of the world to the lion's share (or some even more specific kind of research). If such a change were slow it wouldn't much matter, but if it were fast it could also lead to a decisive strategic advantage, as a company with 0.01% of the world’s resources could find itself with 90% after a drastic reevaluation.

Of course, whether such an abrupt change should happen is again closely related to whether an abrupt change in capacities will happen (though it relies on the further assumption that such a change is unanticipated). Overall I think that it is not too likely, and that the probability can be driven down substantially by clearer communication and decision-making. It still seems like a scenario worth thinking about.

All told, agree with Bostrom that there are a number of reasons to think that very fast changes in capability would lead to a decisive strategic advantage, especially if it came as a surprise to most of the world. Short of rapid changes, I don’t see much reason to think of AI as exceptional.

Regarding evolution I disagree completely with the implicit model of how the future will go or why it might contain morally valuable things. See here for a discussion of my view. On my view, a "singleton" with this more limited capacity would not be very important. Indeed, I'm not even sure what it would mean.

TRIZ-Ingenieur12y00

Diametral opposing to theft and war szenarios you discuss in your paper "Rational Altruist - Why might the future be good?":

How much altruism do we expect?
[...] my median expectation is that the future is much more altruistic than the present.

I fully agree with you and this aspect is lacking in Bostrums book. The FOOM - singleton theory intrinsically assumes egoistic AIs.

Altruism is for me one of the core ingredience towards sustainably incorporating friendly AIs into society. I support your view that the future will be more altruistic tha... (read more)

0Liso12y

We could test it in thought experiment. Chess game human-grandmaster against AI. 1. it is not rapid (not checkmate in begining). We could also suppose one move per year to slow it down. It bring to AI next advantage because it's ability to concentrate so long time. 2. capabilities a) intellectual capabilities we could suppose at same level during the game (if it is played in one day, otherwise we have to think Moore's law) b) human lose (step by step) positional and material capabilities during the game. And it is expected Could we still talk about decisive advantage if it is not rapid and not unexpected? I think so. At least if we won't break the rules.

3diegocaleiro12y

There are two mixed strategies worth noting that facilitate theft. One is propagating and stealing identities. Instead of stealing from a company, you merely become the company wherever it is represented in coded from (name, signature, brand, bank account, online representation etc...). Propagating identities would be the just creating other AI's, subsets of you or changed copies, that by design were made such that they are considered different entities, and therefore you, the AI, cannot be held accountable for their actions. I'd expect Ben Goertzel to have interesting thoughts on this, though no pointers come to mind. The other is mixing theft, production and destruction of coordination. The Italian mafia for instance, did all three. They control legitimate businesses, they control part of the law-enforcement agency (that is, they act as if they were the police or legitimate power) and they destroy crucial nodes of coordination within the legitimate police force. So not only they steal power, but they also make sure that (valuable) coordination is shattered. Promoting anti-coordination is an interesting move in warfare, and I see no reason why an AI would refrain from using it to obtain a decisive strategic advantage, if growth in power and theft were not proving to be enough. I agree with your conditional statements. IF coordination and clear communication THEN low probability of abrupt transformational deleterious change. IF rapid and strategically relevant power obtained THEN it was likely preceded by swift changes in importance of different resources. Our disagreement seems to hinge on the comparative likelihood of those premises, more than different views on how systems work in this context. I'll transfer discussion of evolution - where real disagreement seems to lurk - to your blog to save brainpower from readers here.

See in context