Example of 'decisive strategic advantage': The Spanish in America, 16th century.
Although this example does not present research by a small group, it does show what happens when a small group holds superior technology: A few hundred men conquered a large empire, without resupply or reinforcement from the Old World. This happened at least twice--Cortés and Pizarro--indicating that it was not a fluke.
Well, kind of. The Spaniards succeeded by killing off/holding hostage their opposition's leadership cadre, and through the accidental deployment of biological weapons-they were immune the illnesses that spread from them to the Aztecs, Incas and Mayans.
They won the battle, and they might have won later if they engaged in a full-scale invasion, but it is not completely clear form historical outcomes that they actually had a decisive strategic advantage.
Sure, the diseases were accidental. But beyond that, they used various methods, including hostage-taking and building alliances, and a few hundred men defeated millions, in two cases. Doesn't that show a decisive strategic advantage?
Way more than a few hundred - local rebellions and malcontents joined their cause in droves on the (mistaken) assumption that they would remain in an advantaged state in the aftermath.
I think you are looking at this wrong. Yes, they had help from local rebellions and malcontents. So would an AGI. An AGI taking over the world wouldn't necessarily look like robots vs. humans; it might look like the outbreak of World War 3 between various human factions, except that the AGI was manipulating things behind the scenes and/or acting as a "strategic advisor" to one of the factions. And when the dust settles, somehow the AGI is in charge...
So yeah, I think it really is fair to say that the Spanish managed to conquer empires of millions with just a few hundred men. Twice.
Non-immunity to illnesses is very important to us. Our computers and network infrastructure is more or less immune against script-kiddies and polymorphal viruses and standard attack schemes.
Our systems are not immune against tailored attacks from intelligence agencies or AIs.
Aztecs, Incas and Mayans.
Minor nitpick: Though the Aztecs and Incas were empires that were defeated by the Spanish, the Mayan Empire had long since fallen when the Spanish arrived. There were Mayans, but only in the sense of the ethnic group that remained in small villages in the region that had formerly been a powerful kingdom.
If someone has some money, they can invest it to get more money. Do you know what the difference is between money and intelligence that makes it plausible to expect an abrupt intelligence explosion, but reasonable to expect steady exponential growth for financial investment returns?
One answer is that using your intelligence to to improve your own cognitive architecture is an entirely new field of investment. The economic growth that accrues from modern investing looks steady from inside the modern economy, but it's explosive from the perspective of a premodern society.
One distinction that may matter here is that something is only counted for as money if it is actually traded between distinct agents. If I got 10 000 times better at making houses from scratch and made a house for me but no one else, there would be no increase in the local economy. If I started selling houses, this would be different.
Superintelligence can be thought of as a large island, where there is a lot of internal trade going on that is invisible to the outside economy. If that was counted as value, then the economy would also be exploding. This internal trade opacity that is proper of intelligence and other intra-agent changes seems to be responsible for the distinctive growth extrapolations.
While not exactly investment, consider the case of an AI competing with a human to devise a progressively better high-frequency trading strategy. An AI would probably:
- be able to bear more things in mind at one time than the human
- evaluate outcomes faster than the human
- be able to iterate on its strategies faster than the human
I expect the AI's superior capacity to "drink from the fire hose" together with its faster response time to yield a higher exponent for the growth function than that resulting from the human's iterative improvement.
consider the case of an AI competing with a human to devise a progressively better high-frequency trading strategy.
A more realistic example would be "competing with a human teamed up with a narrow AI".
You're right, that is more realistic. Even so, I get the feeling that the human would have less and less to do as time goes on. I quote:
“He just loaded up on value stocks,” says Mr. Fleiss, referring to the AI program. The fund gained 41% in 2009, more than doubling the Dow’s 19% gain.
As another data point, a recent chess contest between a chess grandmaster (Daniel Naroditsky) working together with an older AI (Rybka, rated ~3050) and the current best chess AI (Stockfish 5, rated 3290) ended with a 3.5 - 0.5 win for Stockfish.
I don't think an article which compares a hedge fund's returns to the Dow (a price-weighted index of about 30 stocks!) can be considered very credible. And there are fewer Quant funds, managing less money, than there were 7 years ago.
One difference is hardware overhang. When new AI technologies are created, many times the amount of hardware they need is available to run them.
However, if an international agreement was reached in advance, and some of the AGI sandbox problems were solved, we might be able to restrict AGI technology to a bounded amount of hardware for a time-if theoretical results which we do not have yet showed that was the appropriate course of action.
We have all kinds of work to do.
There are some qualitative shifts in growth rate with investment. For example, very small amounts of money basically cannot be invested in the financial due to the high fixed costs associated with money laundering regulation, etc.
On the other hand, the more income people have, the more they consume, which reduces growth. Levels of consumption deemed adequate by our ancestors, and indeed consistent with high savings rates, are not viewed as acceptable any more.
Market is more or less stabilized. There are powers and superpowers in some balance. (gain money sometimes could be illusion like bet (and get) more and more in casino).
If you are thinking about money making - you have to count sum of all money in society. If investments means bigger sum of values or just exchange in economic wars or just inflation. (if foxes invest more to hunting and eat more rabbits, there could be more foxes right? :)
In AI sector there is much higher probability of phase-transition (=explosion). I think that's the diference.
How?
Possibility: There could be probably enough HW and we just wait for spark of new algorithm.
Possibility: If we count agriculture revolution as explosion - we could also count massive change in productivity from AI (which is probably obvious).
A team with a billion dollar budget would need to secure something like a 10,000-fold increase in productivity in order to outcompete the rest of the world.
This requirement seems too strong.
First a project doesn't need to monopolize more than half of the world economic throughput to succeed in obtaining a decisive strategic advantage. A small targeted fraction of the economy (say the armies of all countries in the northern hemisphere) may be more than enough. If the pond is chosen correctly, the fish needn't be that big.
Second: a project that is able to efficiently steal at a faster absolute pace than the world economy grows could over the long run dominate it by capturing most created value, without having to go through the honest toil itself.
Third: if we narrow our defition of achievement of a strategic advantage as being merely the ability to stop the behemoth of evolutionary selection pressures at the many levels where they exist - the original intent of the concept of Singleton in papers like "What is a Singleton?" and "The Future of Human Evolution" - then even less would be necessary. A Singleton could be a specialist in it's ability to countenance abrupt changes in evolution - and nothing else - and it would still serve the purpose of avoiding maximal efficiency clans, thereby protecting flamboyant displays and possibly happiness and consciousness on the side.
Note that a 10,000-fold increase would leave you with a modest fraction of total output. A 1,000-fold increase would still leave you smaller than world's militaries.
The theft scenario relies on you being better at theft than the rest of the world. From the outside AI looks more suited to productive activity than war (which also requires big capital investments in personnel and equipment), so I normally think of this as being a counterbalancing factor (that is, it seems significantly more likely that an economically dominant firm would find many of its productive assets confiscated by force, than that a productive firm would begin stealing the world’s resources without serious repercussions). Of course a primary consideration in this discussion is the nature of conflict; in the current world a sufficiently sophisticated AI might fare reasonably well in all-out conflict due to the world being completely unprepared, but I would be quite surprised if that were still the case when the development of human-level AI actually looked plausible. It seems to again come down to the possibility of a rapid and unexpected jump in capabilities.
The most realistic paths along these lines seem to depend on abrupt changes in the importance of different resources. For example, you might think that AI research capacity would change from a small part of the world to the lion's share (or some even more specific kind of research). If such a change were slow it wouldn't much matter, but if it were fast it could also lead to a decisive strategic advantage, as a company with 0.01% of the world’s resources could find itself with 90% after a drastic reevaluation.
Of course, whether such an abrupt change should happen is again closely related to whether an abrupt change in capacities will happen (though it relies on the further assumption that such a change is unanticipated). Overall I think that it is not too likely, and that the probability can be driven down substantially by clearer communication and decision-making. It still seems like a scenario worth thinking about.
All told, agree with Bostrom that there are a number of reasons to think that very fast changes in capability would lead to a decisive strategic advantage, especially if it came as a surprise to most of the world. Short of rapid changes, I don’t see much reason to think of AI as exceptional.
Regarding evolution I disagree completely with the implicit model of how the future will go or why it might contain morally valuable things. See here for a discussion of my view. On my view, a "singleton" with this more limited capacity would not be very important. Indeed, I'm not even sure what it would mean.
There are two mixed strategies worth noting that facilitate theft.
One is propagating and stealing identities. Instead of stealing from a company, you merely become the company wherever it is represented in coded from (name, signature, brand, bank account, online representation etc...). Propagating identities would be the just creating other AI's, subsets of you or changed copies, that by design were made such that they are considered different entities, and therefore you, the AI, cannot be held accountable for their actions. I'd expect Ben Goertzel to have interesting thoughts on this, though no pointers come to mind.
The other is mixing theft, production and destruction of coordination. The Italian mafia for instance, did all three. They control legitimate businesses, they control part of the law-enforcement agency (that is, they act as if they were the police or legitimate power) and they destroy crucial nodes of coordination within the legitimate police force. So not only they steal power, but they also make sure that (valuable) coordination is shattered.
Promoting anti-coordination is an interesting move in warfare, and I see no reason why an AI would refrain from using it to obtain a decisive strategic advantage, if growth in power and theft were not proving to be enough.
I agree with your conditional statements. IF coordination and clear communication THEN low probability of abrupt transformational deleterious change. IF rapid and strategically relevant power obtained THEN it was likely preceded by swift changes in importance of different resources.
Our disagreement seems to hinge on the comparative likelihood of those premises, more than different views on how systems work in this context.
I'll transfer discussion of evolution - where real disagreement seems to lurk - to your blog to save brainpower from readers here.
Diametral opposing to theft and war szenarios you discuss in your paper "Rational Altruist - Why might the future be good?":
How much altruism do we expect?
[...] my median expectation is that the future is much more altruistic than the present.
I fully agree with you and this aspect is lacking in Bostrums book. The FOOM - singleton theory intrinsically assumes egoistic AIs.
Altruism is for me one of the core ingredience towards sustainably incorporating friendly AIs into society. I support your view that the future will be more altruistic than the present: AIs will have more memory to remember behavior of their contacts. The Dunbar's number of social contacts will be higher. Social contacts recognize altruistic behavior and remember this good deed for the future. The wider the social net the higher is the reward for altruistic behavior.
Recent research confirms this perspective: Curry, O., & Dunbar, R. I. M. (2011). Altruism in networks: the effect of connections. Biology Letters, 7(5), 651-653:
The result shows that, as predicted, even when controlling for a range of individual and relationship factors, the network factor (number of connections) makes a significant contribution to altruism, thus showing that individuals are more likely to be altruistic to better-connected members of their social networks.
The idea of AIs and humans monitoring AIs in a constitutional society is not new. Stephen Omohundro presented it in October 2007 at the Stanford EE380 Computer Systems Colloquium on “Self-Improving Artificial Intelligence and the Future of Computing”.
I transcribed part of the Q&A of his talk (starting 51:43)
Q: What about malicious mutations [of the utility function]?
Stephen Omohundro:
Dealing with malicious things is very important. There is an organization - Eliezer is here in the back - he called it the Singularity Institute for Artificial Intelligence, which is trying to ensure that the consequences of these kinds of systems are immune to malicious agents and to accidental unintended consequences. And it is one of the great challenges right now because if you assume that this kind of system is possible and has the kinds of powers we are talking about, it can be useable for great good but also for bad purposes. And so finding a structure which is stable - and I think I agree with Eric [Baum?]- that the ultimate kind of solution that makes more sense to me is essentially have a large ecology of intelligent agents and humans. Such that in a kind of a constitution that everybody follows:
Humans probably will not be able to monitor AIs, because they are thinking faster and more powerfully, but AIs could monitor AIs.
So we set up a structure so that each entity wants to obey the "law", wants to follow the constitution, wants to respect all the various rights that we would to decide on. And if somebody starts violating the law that they have an interest in stopping them from doing that. The hope is that we can create basically a stable future of society with these kinds of entities. The thinking of this is just beginning on that. I think a lot of input is needed from economists, is needed from psychologists, [...] and sociologists [...] as well as computer systems engineers. I mean we really need input from a wide variety of vizpoints.
The FOOM - singleton theory intrinsically assumes egoistic AIs.
No, that's wrong. The speed of takeoff is largely a technical question; from a strategic planning POV, going through a rapid takeoff likely makes sense regardless of what your goals are (unless your friendliness design is incomplete /and/ you have corrigibility aspects; but that's a very special case).
As for what you do once you're done, that does indeed depend on your goals; but forming a singleton doesn't imply egoism or egocentrism of any kind. Your goals can still be entirely focused on other entities in society; it's just that if have certain invariants you want to enforce on them (could be anything, really; things like "no murder", "no extensive torture", "no destroying society" would be unoffensive and relevant examples) - or indeed, more generally, certain aspects to optimize for - it helps a lot if you can stay in ultimate control to do these things.
As Bostrom explains in his footnotes, there are many kinds of singletons. In general, it simply refers to an entity that has attained and keeps ultimate power in society. How much or how little it uses that power to control any part of the world is independent of that, and some singletons would interfere little with the rest of society.
Your argumentation based on the orthogonality principle is clear to me. But even if the utility function includes human values (fostering humankind, preserving a sustainable habitat on earth for humans, protecting humans against unfriendly AI developments, solving the control problem) strong egoistic traits are needed to remain superior to other upcoming AIs. Ben Goertzel coined the term "global AI Nanny" for a similar concept.
How would we get notion of existence of a little interfering FAI singleton?
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
How would we get notion of existence of a little interfering FAI singleton?
The AI's values would likely have to be specifically chosen to get this outcome; something like "let human development continue normally, except for blocking existential catastrophes". Something like that won't impact what you're trying to do, unless that involves destroying society or something equally problematic.
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
Above hypothetical singleton AI would end up either sabotaging the project, or containing the resulting AI. It wouldn't have to stop the UFAI before release, necessarily; with enough of a hardware headstart, later safe containment can be guaranteed fine. Either way, the intervention needn't involve attacking humans; interfering with just the AI's hardware can accomplish the same result. And certainly the development project shouldn't get much chance to fight back; terms like "interdiction", "containment", "sabotage", and maybe "police action" (though that one has unfortunate anthropomorphic connotations) are a better fit than "war".
It seems to again come down to the possibility of a rapid and unexpected jump in capabilities.
We could test it in thought experiment.
Chess game human-grandmaster against AI.
it is not rapid (not checkmate in begining).
We could also suppose one move per year to slow it down. It bring to AI next advantage because it's ability to concentrate so long time.capabilities
a) intellectual capabilities we could suppose at same level during the game (if it is played in one day, otherwise we have to think Moore's law)
b) human lose (step by step) positional and material capabilities during the game. And it is expected
Could we still talk about decisive advantage if it is not rapid and not unexpected? I think so. At least if we won't break the rules.
Before crossover, most of the improvement is coming from the rest of the world. Note that this suggests the rest of the world is also improving similarly fast in that period.
'Decisive strategic advantage' = a level of technological and other advantages sufficient for complete world domination (p78)
Our canonical example, the Manhattan Project, may have been at that level. If the US had chosen to attack Soviet cities with the few atom bombs that they had in the first year, then quickly replenishing their supply quickly while the Soviet Union remained off-balance (scary thought), the US could have dominated the world. Thankfully, they did not, but it does suggest that a superweapon used without moral constraint could have world-dominating effect.
Alternate history is not falsifiable, of course, but that scenario doesn't look all that likely to me. Russia successfully recovered from losing a very large chunk of its territory, a great deal of its army, and most of its manufacturing capacity to Germans in 1941-1942. Losing a few cities (even assuming the bombers could get through -- there were no ICBMs and Russia in 1945 had a pretty good air force and AA capabilities) would not cripple Russia. I would guess that it would just incentivize it to roll over the remainder of Europe. It's not like Stalin ever cared about casualties.
Good point, but I think Bostrom's point about risk aversion does much to ameliorate it. If the US had had a 50% chance of securing global hegemonicy, and a 50% chance of destruction from such a move, it probably would not have done it. A non-risk-averse, non-deontological AI, on the other hands, with its eye on the light cone, might consider the gamble worthwhile.
Probably not: Some nerdy superintelligent AI systems will emerge but humans will try their utmost to shut off early enough. Humankind will become very creative to socialize AGI. The highest risk is that a well funded intelligence agency (e.g. NSA) will be first. Their AI system could make use of TAO knowledge to kill all competing projects. Being nerdy intelligent it could even manipulate competing AI projects that their AIs get "mental" illnesses. This AI will need quite a long time of learning and trust-building until it could take over world dominion.
Bostrums book is a wake-up call. In his book presentation Authors@Google (final six minutes Q&A) he claimed that only half a dozen scientists are working full time on the control problem worldwide. This is by far not enough to cope with future risks. More and effective funding is needed.
Bostrum does not want to destroy his straightforward FOOM-DOOM scenario. He does not discuss technical means of monitoring but only organizational ones.
I fully agree with Bostrum that too few people are working on the control problem. In 2007 Stephen Omohundro asked for synergistic research between psychologists, sociologists and computer system engineers on the control problem. Today we have to conclude that progress is limited.
We have many technical options at hand to prevent that an AI project can obtain decisive strategic advantage:
- Prevent content overhang of powerful inside knowledge (taboos, secrecy, fighting organized internet crime).
- Prevent control access by keeping life supporting infrastructures independent from internet.
- Prevent hardware overhang by improving immunity to cyberattacks (hard crypto).
- Law against backdoors in any system.
- Transparency (AI development, cyberattack monitoring).
- Develop weak-AI with superintelligent capability of monitoring AIs (thought police)
- Develop fixed social conscience utility function.
"What is the best way to push it [risk of doom] down." was Bostrums last sentence at his book presentation. This should be a further point of our discussion.
Well, no life form has achieved what Bostrom calls a decisive strategic advantage. Instead, they live their separate lives in various environmental niches.
One way to look at it is this: Suppose a dominant AGI emerged which was largely running the planet and expanding into the galaxy.
Would it then be impossible to engineer another AGI which survived modestly in some niche or flew off at near the speed of light in a new direction? No.
For the first AGI to be the only AGI, all other AGI development would have to cease without such "niche AGIs" ever being created.
An AGI could be extremely unobtrusive for tens of thousands of years at a time, and even be engaging in some form of self-improvement or replication.
"Sterilizing" matter of all of the "niche AGI" it contains could be quite an involved process.
If you are trying to sterilize, then the level you seek is not only AGI level, but also replicators, which can mobilize lots of resources within a niche.
For the first AGI to be the only AGI, all other AGI development would have to cease without such "niche AGIs" ever being created.
That AGI does not need to stay the only one to solidly stay in power. Since it has been playing the game for longer, it would be reasonable for it to be able to keep tabs on other intelligent entities, and only interfere with their development if they became too powerful. You can still have other entities doing their own thing, there just has to be a predictable ceiling on how much power they can acquire - indeed, that is the idea behind FAI programming: Have the FAI solve some fundamental problems of society, but still leave a society composed of plenty of other intelligences.
This would be made easier if reality is virtualized (i.e. if the singleton AI handles building and maintaining computronium infrastructure, and the rest of society runs as programs using some of the resources it provides); you don't need to monitor every piece of matter for what computations it might carry out, if you've limited how much computation power you give to specific entities, and prevented them from direct write access to physical reality, to begin with.
In the end, I think eventual decisive strategic advantage for a single AI is extremely likely; it's certainly a stable solution, it might happen due to initial timing, and even if doesn't happen right then, it can still happen later. It's far from clear any other arrangement would be similarly stable over the extremely long time horizons of relevance here (which are the same as those for continued existence of intelligences derived from our civilization; in the presence of superintelligent AGIs, likely billions of years). In fact, the most likely alternative to me is that humanity falls into some other existential catastrophe that prevents us from developing AGI at all.
Well, no life form has achieved what Bostrom calls a decisive strategic advantage. Instead, they live their separate lives in various environmental niches.
Ants are probably good example how could organisational intelligence (?) be advantage.
According to wiki ''Ants thrive in most ecosystems and may form 15–25% of the terrestrial animal biomass.''. See also google answer, wiki table or stackexchange.
Although we have to think careful - apex predators does not use to form large biomass. So it could be more complicated to define success of life form.
Problem of humanity is not only global replacer - something which erase all other lifeforms. It could be enough to replace us in our niche. Something which globally (from life viewpoint ) means nothing.
And we dont need to be totally erased to meet huge disaster. Decline population to several milions or tousands... (pets or AI) ... is also unwanted.
We are afraid not decisive strategic advance over ants but over humans.
One possibility to prevent smaller group gain strategic advantage is something like operation Opera.
And it was only about nukes (see Elon Musk statement)...
On selling versions of different AGIs to others, something that does not come up in the text but does in the above paragraphs:
An AGI is a weaponizable technology. Technology capable of running an instance of the code should not be for sale any more than you can buy a tank or an ICBM-it only gets done by international agreement.
Many AGI components should not be for sale individually, just as you cannot buy some parts of a tank, and access to them could be restricted even before they are ever assembled into an AGI or during the time when we only believe they might end up being part of one solution pathway.
Even a remote-controlled minivan is something that would require heavy licensing and ownership restrictions. Measures will be taken so that the owners of self-driving cars have great difficulty in reprogramming the car's control systems.
If we develop an early AI communicating in a slightly stilted way, but with sufficient reasoning and NLP skills to largely replace teachers, accountants, stockbrokers or journalists, then leave it is a shared service like a search engine whose code we never see.
If AGI is used to tell self-driving cars where to drive and robots what to build as part of a master strategy to generate material abundance, maybe the cars can contain enough AI to navigate and pick up cargo, but bundling a complete AGI inside of a device like a car or a robot seems pretty hazardous.
Along with AI containment, the question of which devices get how much reasoning power also needs an answer. One AGI per car or per robot, like we see in sci fi sometimes, seems like a very wrong idea.
However, a singleton with a decisive strategic advantage does not rule out devices which are running separate AGIs. If devices are physically distant, then the speed of light may require such configurations.
Elements of these considerations need additional thought...
Bostrom does not purport to be an expert on international relations, but the chapter does correctly reference one of the critical turning-points of the Cold War-the failure of the Soviet Union to accept the Baruch Plan.
The Baruch Plan would have placed all nuclear power and nuclear weapons technology in the hands of a multilateral organization developed as part of The United Nations. The Soviet Union refused because communist nations were out-voted in the UN.
We can chalk this outcome up to the personality of Stalin, but apparently the Soviets did not feel that they were more secure in a world where nuclear technology was governed multilaterally.
Nations could consider revisiting The Baruch Plan again just for the purpose of governing nuclear weapons. From my perspective, however, we also need to consider a multilateral framework to govern AI and AGI technology.
This framework, which would have similarities and difference from The Baruch Plan, could begin as a means to govern the use of drones and autonomous vehicles in conventional warfare. Drones are going to be used by UN peacekeeping mission, and we should get a jump on selecting the rules.
As the framework is tuned and improved, perhaps it can be expanded. However, strengthening of multilateral institutions will be impossible absent a complete understanding of the perceived interests of all relevant parties.
While The Baruch Plan failed, today we may have time to test frameworks more carefully before they are debated by the international community, and we also have some additional intellectual tools.
Political considerations may dictate that a "managed roll-out" of AI technology is necessary, regardless of when these technologies are first discovered. This is as important a sub-topic of this discourse as any other.
When has anyone gained a 'decisive strategic advantage' at a smaller scale than the world? Can we learn anything interesting about what characteristics a project would need to have such an advantage with respect to the world?
People interested in this particular question may be interested in Peter Thiel's discussions about Last Mover Advantage, available in many media if you google for it.
How useful do you think looking at bombs and space projects (box 5) is for picturing what will happen with AI? (p79-82)
While space projects are more like the Peacock's tail (flamboyant displays, a theme we will return to later in the book), bombs are more like poison indicators (such as colors black red and yellow in snakes).
AI is actually powerful. As Elon puts it, with AI, you are summoning a demon.
Comparing AI with space projects could be beneficial on some dimensions, but it seems to me that comparison with bombs and other military technology, that are more like the ability to poison or summon demons, should take evidential precedence.
On p82 Bostrom argues that even if one research group has access to another's code, it would still take them months to change approach, which could lead to the leaders getting a full FOOM's worth of headstart. But I don't really understand why he thinks this. If I have access to your code, can't I just run it as soon as I notice you're running it? True, you could obfuscate it, or protect your production-box more carefully. But if I have that good monitoring systems, I'm probably a government, in which case I can afford to have a team dedicated to reverse-engineering your approach.
Some reasons that come to mind:
- It might take you a while to come to the conclusion that your technology won't overtake theirs.
- You might have slightly different computational resources, and the code might be specific to that.
Bostrom says the organizational route to superintelligence would require much of the world economy (p83). I don't understand why that would be so. Does someone else?
That seems to depend on the size of the world economy. There may be an absolute minimum of organizational ability for some complex entities to exist - Toby Ord points out that if the human population was much smaller, we wouldn't be able to create the infrastructure that allows for computers.
If the absolute structure of an easy to create Superintelligence requires hundreds of millions of agents in concert, it would appear that a large fraction of the economy is necessary for an observer of a world with a few billion people.
A more obvious note is that the closer you are to representing half the economy, the easier it is to reach the point in which you control 51% of it (which may coincide with strategic advantage in some cases).
Bostrom makes an interesting point that open literature tends to benefit small teams relative to large ones.
Also, he points out that it is easier for a government to nationalise the most successful projects than to ban all projects. The most successful projects are likely to have significant funding, hire well-known experts, go to conferences, etc. Small basement projects, on the other hand, could much more easily escape detection. Unfortunately, this is a pro-arms-race dynamic: it is easy for governments to try to outrace small teams with their own teams than to ban development.
The point about countries defecting from global collaborations is also interesting. It's relatively hard to defect on a nuclear-sharing project: you actually need to build your own reactors, acquire fissile material, etc. But running your own AI project might only requires you to steal some thumb drives.
The "Monitoring" subsection. Bostrom makes good points about why state actors would be likely to seek to monopolize AGI development, if they took it to be a threat to their power monopolies. I hadn't given the possibility sufficient consideration until this point. These actors do seem disinterested so far, but (as Bostrom points out) that might well be a pretense, and even if not it doesn't mean they'll remain so for too long to matter.
Bostrum underestimates complexity of learning, compare Robin Hanson's criticism "I Still Don’t Get Foom" on his book.
Assume following small team scenario that could reach a decisive advantage: A hedge fond seeks world dominion and develops in secrecy a self-improving AI. Following skills shall reach superhuman capabilities:
- cyberattack and cryptanalysis
- semantic comprehension of tech and business documents
- trading strategy
Latest when this AI reaches a decisive strategic advantage over other market players they will acknowledge this instantly. Insider trade investigations will soon reveal that this hedge fond was breaking the law.
This AI had not yet the time to learn all other skills needed for world dominion:
- creating a globally operating robot guard
- creativity to develop countermeasures against all thinkable switching-off scenarios
- influencing humans (politicians, military brass and mass propaganda).
A human is capable to run a large company or a government earliest at the age of 30. To learn how to influence people has very few speedup options and books are of little help. The AI has to acquire real insight comprehension of human values and motivations to become a successful world leader.
A quick-and-dirty AI without carefully designed utility function and wisdom from humane values will evoke the utmost available power of the entire world united to switch off this AI.
A human is capable to run a large company or a government earliest at the age of 30.
Yes indeed I am convinced that 30 years of learning is a minimum for running a large company or a government. I compiled data from 155 government leaders of five countries. On the average they took office for their first term at the age of 54.3 years.
For my above statement allow me to substract 2 standard deviations (2 x 8.5 = 19 years). A government leader is therefore with 97.7% probability older than 35.3 years when he takes office for the first time. The probability of a government leader being younger than 30 years is 0.22%, calculated from the standard distribution. William Pitt the Younger became youngest UK Prime Minister in 1783 at the age of 24. He is the only one younger than 30 in my data set.
Lumifer, could you be so kind to compile a similar statistical evaluation of about top 150 companies of these five countries. I can help you with DAX 30. I am expecting that the average will be lower for following reasons:
- Trustbuilding is more time consuming in politics than in business and involves more people.
- Some startup pioneers started very young and their companies grew extremely quick.
- Right tail of age distribution >65 years will be thin for companies.
After this we both know who of us might LOL. We both know that a doom scenario is a possible outcome of AI development. My intention is to understand what knowledge is needed to rule the world and how it is possible to hide this content to slow down the learning curve of an AI capable of taking over everything.
I compiled data
Your data shows what typically happens. That's a bit different from what "a human is capable" of.
Technically speaking, a single counter-example overturns your claim.
This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.
Welcome. This week we discuss the seventh section in the reading guide: Decisive strategic advantage. This corresponds to Chapter 5.
This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.
There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).
Reading: Chapter 5 (p78-91)
Summary
5. Disagreement. Note that though few people believe that a single AI project will get to dictate the future, this is often because they disagree with things in the previous chapter - e.g. that a single AI project will plausibly become more capable than the world in the space of less than a month.
If you are particularly interested in these topics, and want to do further research, these are a few plausible directions, some inspired by Luke Muehlhauser's list, which contains many suggestions related to parts of Superintelligence. These projects could be attempted at various levels of depth.
How to proceed
This has been a collection of notes on the chapter. The most important part of the reading group though is discussion, which is in the comments section. I pose some questions for you there, and I invite you to add your own. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!
Next week, we will talk about Cognitive superpowers (section 8). To prepare, read Chapter 6. The discussion will go live at 6pm Pacific time next Monday 3 November. Sign up to be notified here.