Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.
Any scenario where advanced AI takes over the world requires some mechanism for an AI to leverage its position as ethereal resident of a computer somewhere into command over a lot of physical resources.
One classic story of how this could happen, from Eliezer:
- Crack the protein folding problem, to the extent of being able to generate DNA strings whose folded peptide sequences fill specific functional roles in a complex chemical interaction.
- Email sets of DNA strings to one or more online laboratories which oﬀer DNA synthesis, peptide sequencing, and FedEx delivery. (Many labs currently oﬀer this service, and some boast of 72-hour turnaround times.)
- Find at least one human connected to the Internet who can be paid, blackmailed, or fooled by the right background story, into receiving FedExed vials and mixing them in a specified environment.
- The synthesized proteins form a very primitive “wet” nanosystem which, ribosomelike, is capable of accepting external instructions; perhaps patterned acoustic vibrations delivered by a speaker attached to the beaker.
- Use the extremely primitive nanosystem to build more sophisticated systems, which construct still more sophisticated systems, bootstrapping to molecular nanotechnology—or beyond.
You can do a lot of reasoning about AI takeover without any particular picture of how the world gets taken over. Nonetheless it would be nice to have an understanding of these possible routes. For preparation purposes, and also because a concrete, plausible pictures of doom are probably more motivating grounds for concern than abstract arguments.
So MIRI is interested in making a better list of possible concrete routes to AI taking over the world. And for this, we ask your assistance.
What are some other concrete AI takeover mechanisms? If an AI did not have a solution to the protein folding problem, and a DNA synthesis lab to write off to, what else might it do?
We would like suggestions that take an AI from being on an internet-connected computer to controlling substantial physical resources, or having substantial manufacturing ability.
We would especially like suggestions which are plausible given technology that normal scientists would expect in the next 15 years. So limited involvement of advanced nanotechnology and quantum computers would be appreciated.
We welcome partial suggestions, e.g. 'you can take control of a self-driving car from the internet - probably that could be useful in some schemes'.
[link] Nick Beckstead on improving disaster shelters to increase the chances of recovery from a global catastrophe
What is the problem? Civilization might not recover from some possible global catastrophes. Conceivably, people with access to disaster shelters or other refuges may be more likely to survive and help civilization recover. However, existing disaster shelters (sometimes built to ensure continuity of government operations and sometimes built to protect individuals), people working on submarines, largely uncontacted peoples, and people living in very remote locations may serve this function to some extent.
What are the possible interventions? Other interventions may also increase the chances that humanity would recover from a global catastrophe, but this review focuses on disaster shelters. Proposed methods of improving disaster shelter networks include stocking shelters with appropriately trained people and resources that would enable them to rebuild civilization in case of a near-extinction event, keeping some shelters constantly full of people, increasing food reserves, and building more shelters. A philanthropist could pay to improve existing shelter networks in the above ways, or they could advocate for private shelter builders or governments to make some of the improvements listed above.
Who else is working on it? Some governments maintain bunkers in order to maintain continuity of government and/or to protect their citizens. Some individuals purchase and maintain private disaster shelters.
Questions for further investigation: With the possible exception of pandemic specifically engineered to kill all humans, I am aware of no scenario in which improved disaster shelters would plausibly enable a small group of people to survive a sudden near-extinction event. In the case of other catastrophes where a much larger number of people would survive, I would guess that improved refuges would play a relatively small role in helping humanity to recover because they would represent a small share of relevant people and resources. Many challenging questions about improving refuges remain, but I would prioritize investigating other issues at this point because refuges seem likely to be of limited value and alternative strategies (such as improving biosecurity and increasing the resilience of industrial and agricultural systems) seem more likely to effectively reduce the global catastrophic risks that improving refuges might plausibly address.
I’ve seen Joseph Tainter’s “The Collapse of Complex Societies” recommended in a few different places. Jared Diamond’s book might be one of them, the guest-posts of Captain David Ryan aka “Tony Comstock” for James Fallows at the Atlantic might be another. The sidebar of John Robb’s “Global Guerrillas” blog is the only one I remember with certainty. It’s a not a very long book, and you can get the gist of it from Tainter’s wikipedia page.
Lots of people have found civilizational collapses to be interesting, and Tainter reviews many of their theories while finding them wanting. The “eleven major themes in the explanation of collapse” he lists are depletion/cessation of a vital resource, establishment of a new resource base (which I found too stupid to take seriously even momentarily), insurmountable catastrophe, insufficient response to circumstances (which is almost tautological), other complex societies, intruders, class conflict or elite mismanagement, social dysfunction, mystical factors, chance concatenation of events (almost tautological if you don’t think collapse is predetermined) and economic factors. Like Tainter, I find the “mystical” theories to not really constitute theories at all, although some of the most popular writers on the subject (Spengler, Toynbee, various ancients) are included there. Tainter often contrasts “integrative” (or “functional”) theories on the origin of the state/complexity vs “conflict” theories, and acknowledges that he is more partial toward the former. Unfortunately, most of the latter theorists he lists are Marxists and carry a lot of baggage. The observation that throughout much of history some set of people ruled over others as a result of military victory regardless of any benefit to the subjects (though a Leviathan may happen to have upsides) predates Marx, with Ibn Khaldun being one of the few non-Marxist examples Tainter mentions. That’s not to say Tainter is anti-Marxism, he actually compares Marxist “social science” to Einsteinian physics and Darwinian biology! I suppose there is (or was) just such a heavy representation of Marxists among academic anthropologists and historians that Tainter regards Marxism as somewhat normative, whereas to me it’s something weird and laughable like Holocaust revisionism.
Resource depletion is the reverse of the theory I found so absurd, and (showing there is hope for humanity) it is a much more popular theory. J. Donald Hughes blamed Rome’s collapse in part on deforestation, but W. Groenman van Waateringe some years later provided evidence at the time that cereal pollens declined while forest pollens increased. That of course is not a causal proof, since it is documented that when the empire was declining many agricultural regions became depopulated. Waateringe blames agricultural intensification for increasing the population and thus the demands on agriculture, but to me that just raises the question of why marginal agricultural lands weren’t reclaimed. There actually is an explanation for that depopulation, but like Tainter I’m not going to get to that in a hurry. Tainter finds this theory (like most other theories he rejects) unsatisfactory because complex societies should have leaders who notice the depletion and think of a response. I am reminded of David St. Hubbin’s girlfriend in Spinal Tap who says “It’s just a problem! It get’s solved!” Sometimes a solution is not within a society’s feasible choice-set. Tainter briefly acknowledges that possibility (noting that it would have to be proved, which is difficult given how little information we have about many ancient societies) but spends more time castigating imaginary opponents depicted as claiming societal elites just stood around slack-jawed rather than attempting to deal with the situation. I would call that a strawman, except that Jared Diamond’s “Collapse” bears too much resemblance. He also mentions Richard Wilkinson’s documenting that deforestation spurred development in late/post medieval England, which is really just extra evidence that Hughes was wrong (as had already been mentioned) rather than a broader point against a class of theories. I should acknowledge that Tainter also cites evidence on the differential abandonment of cities and the failure to correlate with expected environmental characteristics, which is just the sort of thing that would later puncture Jared Diamond’s take on the Maya. His point that greed is constant enough (or its variance poorly enough understood) to make it a poor explanation of a variable situation is fair enough, but he can’t dismiss theories of collapse based on mismanagement because by their nature they should keep their society going, even if only out of self-interest. There are basic agency problems that mean one shouldn’t identify the interests of elite (or non-elite, for that matter) actors with that of a larger organization. In an uncertain world it also makes sense to discount the future (you or your dynasty might be replaced, might as well get what you can while you can).
Steve Sailer once critiqued Diamond’s thesis by noting that societies tend to die from homicide rather than suicide, and lumping together two of Tainter’s rejected explanations would make for a very popular theory. Tainter, however, would exclude most cases clearly caused by another complex society because those involve absorbtion rather than collapse (indigenous populations thoroughly devastated by disease before Europeans even arrived would be exceptions). So his question is then why a complex society would succumb to less complex intruders. Sometimes it may not be so easy to disentangle the two scenarios, such as when the persian & eastern roman empires exhausted themselves fighting each other, leaving themselves open to the Muslim invaders exploding out of Arabia (although it was only the persians that succumbed in fairly short order, and Tainter wouldn’t consider that a collapse). Tainter says it is “unsatisfactory [...] that a recurrent process – collapse – is explained by a random variable, by historical accident”. If random numbers for that variable (I’m imagining a stochastic process with a threshold for collapse rather than a binary control variable) are constantly being generated over time, it shouldn’t be that unsatisfactory that they recur throughout history. Tainter does make the legitimate point that elites, with a number of Roman emperors being good examples, have often proved capable of dealing with barbarian intrusions. But there’s no guarantee that they will always be successful. He also wonders why invaders would “destroy those things which repay conquest”. The obvious answer is that, by the second law of thermodynamics, it’s very easy to break things, and that includes during the process of conquering & looting. Some relatively sophisticated barbarians may conquer a territory and leave much of the administrative apparatus intact to rule as before, others may have no particular interest (or competence) in being bound to a territory and collecting scraps of taxes from farmers.
The collapse of Rome is probably the most famous example (at least to westerners) and forms one of his three case studies, paradigmatic of the most complex sort of society to collapse. I find it more enlightening than the others because, and call me a drunk looking under a lamppost if you will, it’s the most well documented. Among the things documented is that the proximate cause of collapse was invasion by various (mostly Germanic) barbarians. That is discussed extensively in Peter Heather’s “Empires and Barbarians” which I have discussed earlier. It’s because I read that one so recently that a few of Tainter’s remarks stuck out. Focusing on the internal soundness of a society and its affordable scale/complexity he writes “The Germanic kingdoms that succeeded Roman rule in the West were more successful at resisting invasions”. If he limited that claim to the particular kingdoms which survived past the dark ages, it would be rather tautological. But if he means germanic kingdoms generally, then it just doesn’t seem to be the case. They got invaded and replaced all the time, we just don’t remember the ones that died out. Part of the reason the Romans had such problems with barbarians is that one group would get invaded by another, and then start moving around and displacing other barbarians. The western Roman empire, in contrast, was able to survive many invasions before the last Roman emperor was toppled. Tainter portrays the formation of the empire as a process in which a territory is able to summon the resources to mobilize a force to conquer more territory to extract its resources, then rinse and repeat in a self-sustaining cycle until it expanded too far to get many marginal returns. There is some truth to that, but it overlooks the non-extractive aspect of Roman rule which increased productivity in conquered territories, thereby making those territories more attractive as a target for raiding. In Heather’s story, barbarian confederations on the border engaged in a process of competitive selection for the strength to hold an attractive position (for reasons of trade, raiding and diplomatic subsidy) and eventually the size and cohesion necessary to survive and settle within Roman territory. Focusing on the internals of the collapsed societies, he overlooks any dynamics occurring within outside societies that could give them the capability of defeating the imperial power. Heather’s account is similar to Peter Turchin’s in “War and Peace and War” except that, like Khaldun, Turchin focuses more on the softening effects of metropolitan decadence that renders old dynasties vulnerable to the hardened asabiya-endowed border marchers.
Tainter’s two other case studies are the Mayan lowland citystates and Chaco canyon cliff-dwellers. The Mayans are less complex (or at least less well-documented, since the conquistadors destroyed many of the remaining documents) and the Chacoans even less so. He also used the Ik as an example of an extremely simple society that collapsed even further below the level of familial organization, but he didn’t discuss it all that much and I’m not sure how reliable primary source Colin Turnbull was (supposedly they hadn’t been hunter-gatherers for centuries when their supposed “livelihood” of hunting was banned). The interesting thing about the Maya is that there were multiple relatively equivalent city-states rather than one dominant hegemon. Tainter includes them as a case study of collapse, even as he states elsewhere that collapse is not an option for “peer polity competition” because the weakening of one peer just invites conquest by another. Also, rather than devoting most of their resources under duress to a standing army (something documented in the Roman case) Tainter discusses the building of monuments as conspicuous consumption to demonstrate how powerful and brutal (per the depictions of torture) the city was, rather similar to the story Diamond tells. I don’t know what kind of evidence we have for the scale of their military expenditures, although we know they did war from time to time. There was no writing whatsoever in Chaco canyon, so we are left with the old archeological standby of potsherds and whatnot. Tainter does make the interesting point that the culture benefitted from uniting different ecological niches, with higher elevation territories having more agricultural productivity in cold wet years while lower elevation ones were more productive in warm dry ones. An economist would say that this diversified portfolio allowed for more consumption smoothing. However, I was confused by Tainter’s argument that as more outlier territories were incorporated diversity and gains from exchange went down. As long as the ratio between high and low places was stable, incorporating more territories should not cause any problems in that respect. Admittedly, this does mean that there are more viable subsets of communities that would be individually stable if they withdrew, which is indeed what he claims happened eventually. But he also seemed to be suggesting that the system overally was degrading its performance, without clearly stating whether an excess of a particular type of environment was upsetting the balance.
Tainter’s theory to explain collapse is declining marginal returns. This is a common concept in economics, but it is normally used to understood how equilibrium can develop. Applied to a society, we would expect the declining returns to territorial expansion or administrative complexity (the former often requiring some degree of the latter) to result in eventual stasis rather than collapse. David Friedman has an interesting paper on how the advantages of taxing trade, land or labor gave rise to different equilibria for the sizes & boundaries of polities during the Roman, medieval and nationalist eras in Europe. In Friedman’s theory, each shift between eras resulted from some exogenous change rather being part of the internal logic of societies. Tainter relates some various interesting bits from C. Northcote Parkinson’s “Parkinson’s Law, and Other Studies in Administration”. For example, while “between 1914 and 1967, the number of capital ships in the British Navy declined by 78.9 percent, the number of officers and enlisted men by 32.9 percent, and the number of dockyard workers by 33.7 percent [...] the number of dockyard officials and clerks increased by 247 percent, and the number of Admiralty officals by 767 percent” (emphasis added). Mencius Moldbug would not be surprised to learn that “between 1935 and 1954 the number of officals in the British Colonial office increased by 447 percent” even though “the empire administered by these officials shrank considerably”. These examples are important because they do not demonstrate an increasingly large requirement of administrators for a marginal increase in size/complexity of an entity to be administered, but paying more for less. Parkinson’s explanation was bureaucratic self-serving, which Tainter rejects because he finds trends of increasing hierarchical specialization in the private sector. But because Tainter fails to distinguish between declining marginal returns (eventually reaching zero at a steady-state) and NEGATIVE returns he doesn’t specify whether the latter occurs in the private sector (though Karl Smith would not be surprised if it does for many publicly owned corporations whose shareholders would be better served by liquidation of assets). The growth of administration in higher education would also count, but as a heavily subsidized non-profit sector I can’t say it would qualify. At one point Tainter acknowledges “In many cases this increased, more costly complexity will yield no increased benefits, at other times the benefits will not be proportionate to costs” (emphasis in the original). This is precisely the question at issue of elite mismanagement or the out-of-control inertia of expanding administrative bureaucracies, but as noted he rejected Parkinson’s theory and mocks the idea of societies as runaway trains as self-evidently absurd. Instead he portrays collapse as a choice which is preferable once marginal returns have declined to a certain point. This didn’t entirely make sense to me, because if a society has accidentally shot part the point of zero marginal returns to one of negative returns, the sensible thing is just to reduce that marginal increase in complexity to return to the steady state with zero marginal returns.
The Roman empire sometimes seemed to behave in such a manner, losing some territory and sticking with a more defensible and adminstrable domain (although in Heather’s account some of the lost territory was among the most agriculturally productive), although Tainter thinks the conquests of Britain & Dacia never paid for themselves. So why the path dependency so that changes are not simply reversible? There could be consumption of a not easily renewable resource, a sort of borrowing from the future that leaves future generations deeper in the red. This could happen with soil deterioration, though Tainter doesn’t discuss that much (odd, despite his focus on societies as means of managing sources of energy). His example of Roman emperors increasingly resorting to the debasing of the currency could count (by Diocletian’s time it collected taxes in kind rather than the currency it had rendered nearly worthless), as well as the selling of imperial land. The larger problem in Rome seemed to be an increasingly large portion of subjects who were citizens (both urban proletariat and squabbling elites) subject to fewer or no taxes, while marginal lands were abandoned by overtaxed farmers. An odd feature of the empire was that election officials had to cover the costs of their own office, and as expenses rose there were fewer wealthy people willing to come forward as candidates, until the position was made hereditary. It became obligatory to farm certain deserted lands, with peasants drafted by local city Senates, and Constantine made soldiery a hereditary profession (which required a number of new laws over time to deal with sons who’d rather not follow that career). Taxation of land was simplistic and did not vary based on its quality or yield, so a farmer of marginal land would often be better off working for the owner of a more productive territory and paying rent than failing to cover the taxes on his own plot. With agricultural labor becoming legally tied to the land, we can see the clear beginnings of serfdom and the manorial system. As mentioned, Tainter views the Roman collapse as a choice (as he does others), although of course accounts from the time were more apt to regard it as unfortunate failure or divine punishment.
Interestingly, the “peer polity competition” that replaced Roman civilization is a situation he regards as invulnerable to collapse as opposed to absorbtion, and by removing that “option” he thinks this made peasants demand democratic representation. He acknowledges that this did not happen in the “Warring States” period of China, and instead the Confucian ideology of governance developed. He suggests “Perhaps participatory governance was simply not possible in ancient societies that were so much larger, demographically and territorially, than the Greek city-states”. Someone should have told James Madison (and I’m not being sarcastic). Interestingly enough, there was a civilization of Greek city states which did collapse, just as we’ve mentioned the lowland Maya doing. These are the Mycenaean Greeks who preceded the Dark Ages of Homer’s time. Their collapse is usually attributed to invasion by Dorian Greeks, but Tainter isn’t convinced there’s enough evidence for the Dorians’ presence. Because “Collapse occurs, and can only occur, in a power vacuum” (emphasis in the original) both the Mycenaean Greek and lowland Maya polities must have experienced simultaneous collapse.
The choice of peasants may be limited to passively withdrawing support and just not working very hard (I’d have more to say on that if I’d read James Scott’s “Weapons of the Weak”), but even then I don’t think it’s a desirable outcome for peasants. I’ve mentioned Heather on the greater productivity of Roman territory, and what do you think happened to the masses when that productivity crashed? My understanding of the current archaelogical consensus is that the population crashed as well. Tainter talks about the malnourished skeletons of the peasantry as evidence for the undesirability of certain degrees of complexity, but we also know that English peasants ate better after so many of their peers died of the bubonic plague (it’s also known that peasants have poorer diets than hunter-gatherers, though in Darwinian terms you definitely want to be a farmer). As in James Scott’s account of highland southeast asia (which I don’t entirely buy) was there much cultural defection of the peasantry to the greener pastures outside civilization? Tainter writes that “In 378 [...] Balkan miners went over en masse to the Visigoths”, and that others wished to be conquered/liberated. It is precisely due to the risk of being conquered that he argues is the reason many societies don’t simply revert to a lower level of complexity “even if marginal returns are unfavorable”, but it’s unclear whether conquest is one of outcomes being factored into those marginal returns.
Few people are going to read this book without speculating on their own complex society’s liability to collapse. John Robb and James Kunstler (along with some others in the “Peak Oil” camp) are going to place a high probability on it, while the Singularitarians have the opposite view. Globalization could mean the entire world is now in a state of “peer polity competition” but modern norms (and economic incentives) against conquest and giving war a chance means “failed states” can keep failing for a long time without someone replacing the bad management. Tainter’s studied societies are also Malthusian agricultural ones, it’s hard to know if the same logic will generalize past the industrial revolution. In modern technological economies the costs and benefits of advances may not be simple increasing or declining curves. Robin Hanson doesn’t even consider nearly free energy (which would very important to Tainter) to be nearly as important as the replacement of most human labor by computers (since the latter takes up so much more of GDP). When Tainter was writing there was still just the slightest possibility of nuclear armageddon, now the most likely candidates for death by complexity are grey goo or an unstoppable manmade pandemic. My two cents are that collapse is unlikely in my lifetime, and that’s for the better considering how much worse things could be.
Don't let your minds be killed, but I was wondering if there were any existential risk angles to the coming American election (if there isn't, then I'll simply retreat to raw, enjoyable and empty tribalism).
I can see three (quite tenuous) angles:
- Obama seems more likely to attempt to get some sort of global warming agreement. While not directly related to Xrisks per se, this would lead to better global coordination and agreement, which improves the outlook for a lot of other Xrisks. However, pretty unlikely to succeed.
- I have a mental image that Republicans would be more likely to invest in space exploration. This is a lot due to Newt Gingrich, I have to admit, and to the closeness between civilian and military space projects, the last of which are more likely to get boosts in Republican governments.
- If we are holding out for increased population rationality as being a helping factor for some Xrisks, then the fact the the Republicans have gone so strongly anti-science is certainly a bad sign. But on the other hand, its not clear whether them winning or losing the election is more likely to improve the general environment for science among their supporters.
But these all seem weak factors. So, less wronger, let me know: are the things I should care about in the election, or can I just lie back and enjoy it as a piece of interesting theatre?
In this essay I argue the following:
Brain emulation requires enormous computing power; enormous computing power requires further progression of Moore’s law; further Moore’s law relies on large-scale production of cheap processors in ever more-advanced chip fabs; cutting-edge chip fabs are both expensive and vulnerable to state actors (but not non-state actors such as terrorists). Therefore: the advent of brain emulation can be delayed by global regulation of chip fabs.
Full essay: http://www.gwern.net/Slowing%20Moore%27s%20Law
I first attempted to post this in 2009, but bounced off the karma wall. Since then, MY forgetfulness and procrastination have been its nemesis.
I invite you to listen (read) in an unusual way. "Consider it": think WITH this idea for a while. There will be plenty of time to refute it later. I find that, if I START with, "That's so wrong!", I really weaken my ability to "pan for the gold".
Remember the Swamp!
I looked over the tag cloud and didn't see:
- Existential Risk
- Nuclear arsenals