TGGP a frequent commenter at Overcoming Bias (and hence old LessWrong), writes about his thoughts on a book by Joseph Tainter.

I’ve seen Joseph Tainter’s “The Collapse of Complex Societies” recommended in a few different places. Jared Diamond’s book might be one of them, the guest-posts of Captain David Ryan aka “Tony Comstock” for James Fallows at the Atlantic might be another. The sidebar of John Robb’s “Global Guerrillas” blog is the only one I remember with certainty. It’s a not a very long book, and you can get the gist of it from Tainter’s wikipedia page.

Lots of people have found civilizational collapses to be interesting, and Tainter reviews many of their theories while finding them wanting. The “eleven major themes in the explanation of collapse” he lists are depletion/cessation of a vital resource, establishment of a new resource base (which I found too stupid to take seriously even momentarily), insurmountable catastrophe, insufficient response to circumstances (which is almost tautological), other complex societies, intruders, class conflict or elite mismanagement, social dysfunction, mystical factors, chance concatenation of events (almost tautological if you don’t think collapse is predetermined) and economic factors. Like Tainter, I find the “mystical” theories to not really constitute theories at all, although some of the most popular writers on the subject (Spengler, Toynbee, various ancients) are included there. Tainter often contrasts “integrative” (or “functional”) theories on the origin of the state/complexity vs “conflict” theories, and acknowledges that he is more partial toward the former. Unfortunately, most of the latter theorists he lists are Marxists and carry a lot of baggage. The observation that throughout much of history some set of people ruled over others as a result of military victory regardless of any benefit to the subjects (though a Leviathan may happen to have upsides) predates Marx, with Ibn Khaldun being one of the few non-Marxist examples Tainter mentions. That’s not to say Tainter is anti-Marxism, he actually compares Marxist “social science” to Einsteinian physics and Darwinian biology! I suppose there is (or was) just such a heavy representation of Marxists among academic anthropologists and historians that Tainter regards Marxism as somewhat normative, whereas to me it’s something weird and laughable like Holocaust revisionism.

Resource depletion is the reverse of the theory I found so absurd, and (showing there is hope for humanity) it is a much more popular theory. J. Donald Hughes blamed Rome’s collapse in part on deforestation, but W. Groenman van Waateringe some years later provided evidence at the time that cereal pollens declined while forest pollens increased. That of course is not a causal proof, since it is documented that when the empire was declining many agricultural regions became depopulated. Waateringe blames agricultural intensification for increasing the population and thus the demands on agriculture, but to me that just raises the question of why marginal agricultural lands weren’t reclaimed. There actually is an explanation for that depopulation, but like Tainter I’m not going to get to that in a hurry. Tainter finds this theory (like most other theories he rejects) unsatisfactory because complex societies should have leaders who notice the depletion and think of a response. I am reminded of David St. Hubbin’s girlfriend in Spinal Tap who says “It’s just a problem! It get’s solved!” Sometimes a solution is not within a society’s feasible choice-set. Tainter briefly acknowledges that possibility (noting that it would have to be proved, which is difficult given how little information we have about many ancient societies) but spends more time castigating imaginary opponents depicted as claiming societal elites just stood around slack-jawed rather than attempting to deal with the situation. I would call that a strawman, except that Jared Diamond’s “Collapse” bears too much resemblance. He also mentions Richard Wilkinson’s documenting that deforestation spurred development in late/post medieval England, which is really just extra evidence that Hughes was wrong (as had already been mentioned) rather than a broader point against a class of theories. I should acknowledge that Tainter also cites evidence on the differential abandonment of cities and the failure to correlate with expected environmental characteristics, which is just the sort of thing that would later puncture Jared Diamond’s take on the Maya. His point that greed is constant enough (or its variance poorly enough understood) to make it a poor explanation of a variable situation is fair enough, but he can’t dismiss theories of collapse based on mismanagement because by their nature they should keep their society going, even if only out of self-interest. There are basic agency problems that mean one shouldn’t identify the interests of elite (or non-elite, for that matter) actors with that of a larger organization. In an uncertain world it also makes sense to discount the future (you or your dynasty might be replaced, might as well get what you can while you can).

Steve Sailer once critiqued Diamond’s thesis by noting that societies tend to die from homicide rather than suicide, and lumping together two of Tainter’s rejected explanations would make for a very popular theory. Tainter, however, would exclude most cases clearly caused by another complex society because those involve absorbtion rather than collapse (indigenous populations thoroughly devastated by disease before Europeans even arrived would be exceptions). So his question is then why a complex society would succumb to less complex intruders. Sometimes it may not be so easy to disentangle the two scenarios, such as when the persian & eastern roman empires exhausted themselves fighting each other, leaving themselves open to the Muslim invaders exploding out of Arabia (although it was only the persians that succumbed in fairly short order, and Tainter wouldn’t consider that a collapse). Tainter says it is “unsatisfactory [...] that a recurrent process – collapse – is explained by a random variable, by historical accident”. If random numbers for that variable (I’m imagining a stochastic process with a threshold for collapse rather than a binary control variable) are constantly being generated over time, it shouldn’t be that unsatisfactory that they recur throughout history. Tainter does make the legitimate point that elites, with a number of Roman emperors being good examples, have often proved capable of dealing with barbarian intrusions. But there’s no guarantee that they will always be successful. He also wonders why invaders would “destroy those things which repay conquest”. The obvious answer is that, by the second law of thermodynamics, it’s very easy to break things, and that includes during the process of conquering & looting. Some relatively sophisticated barbarians may conquer a territory and leave much of the administrative apparatus intact to rule as before, others may have no particular interest (or competence) in being bound to a territory and collecting scraps of taxes from farmers.

The collapse of Rome is probably the most famous example (at least to westerners) and forms one of his three case studies, paradigmatic of the most complex sort of society to collapse. I find it more enlightening than the others because, and call me a drunk looking under a lamppost if you will, it’s the most well documented. Among the things documented is that the proximate cause of collapse was invasion by various (mostly Germanic) barbarians. That is discussed extensively in Peter Heather’s “Empires and Barbarians” which I have discussed earlier. It’s because I read that one so recently that a few of Tainter’s remarks stuck out. Focusing on the internal soundness of a society and its affordable scale/complexity he writes “The Germanic kingdoms that succeeded Roman rule in the West were more successful at resisting invasions”. If he limited that claim to the particular kingdoms which survived past the dark ages, it would be rather tautological. But if he means germanic kingdoms generally, then it just doesn’t seem to be the case. They got invaded and replaced all the time, we just don’t remember the ones that died out. Part of the reason the Romans had such problems with barbarians is that one group would get invaded by another, and then start moving around and displacing other barbarians. The western Roman empire, in contrast, was able to survive many invasions before the last Roman emperor was toppled. Tainter portrays the formation of the empire as a process in which a territory is able to summon the resources to mobilize a force to conquer more territory to extract its resources, then rinse and repeat in a self-sustaining cycle until it expanded too far to get many marginal returns. There is some truth to that, but it overlooks the non-extractive aspect of Roman rule which increased productivity in conquered territories, thereby making those territories more attractive as a target for raiding. In Heather’s story, barbarian confederations on the border engaged in a process of competitive selection for the strength to hold an attractive position (for reasons of trade, raiding and diplomatic subsidy) and eventually the size and cohesion necessary to survive and settle within Roman territory. Focusing on the internals of the collapsed societies, he overlooks any dynamics occurring within outside societies that could give them the capability of defeating the imperial power. Heather’s account is similar to Peter Turchin’s in “War and Peace and War” except that, like Khaldun, Turchin focuses more on the softening effects of metropolitan decadence that renders old dynasties vulnerable to the hardened asabiya-endowed border marchers.

Tainter’s two other case studies are the Mayan lowland citystates and Chaco canyon cliff-dwellers. The Mayans are less complex (or at least less well-documented, since the conquistadors destroyed many of the remaining documents) and the Chacoans even less so. He also used the Ik as an example of an extremely simple society that collapsed even further below the level of familial organization, but he didn’t discuss it all that much and I’m not sure how reliable primary source Colin Turnbull was (supposedly they hadn’t been hunter-gatherers for centuries when their supposed “livelihood” of hunting was banned). The interesting thing about the Maya is that there were multiple relatively equivalent city-states rather than one dominant hegemon. Tainter includes them as a case study of collapse, even as he states elsewhere that collapse is not an option for “peer polity competition” because the weakening of one peer just invites conquest by another. Also, rather than devoting most of their resources under duress to a standing army (something documented in the Roman case) Tainter discusses the building of monuments as conspicuous consumption to demonstrate how powerful and brutal (per the depictions of torture) the city was, rather similar to the story Diamond tells. I don’t know what kind of evidence we have for the scale of their military expenditures, although we know they did war from time to time. There was no writing whatsoever in Chaco canyon, so we are left with the old archeological standby of potsherds and whatnot. Tainter does make the interesting point that the culture benefitted from uniting different ecological niches, with higher elevation territories having more agricultural productivity in cold wet years while lower elevation ones were more productive in warm dry ones. An economist would say that this diversified portfolio allowed for more consumption smoothing. However, I was confused by Tainter’s argument that as more outlier territories were incorporated diversity and gains from exchange went down. As long as the ratio between high and low places was stable, incorporating more territories should not cause any problems in that respect. Admittedly, this does mean that there are more viable subsets of communities that would be individually stable if they withdrew, which is indeed what he claims happened eventually. But he also seemed to be suggesting that the system overally was degrading its performance, without clearly stating whether an excess of a particular type of environment was upsetting the balance.

Tainter’s theory to explain collapse is declining marginal returns. This is a common concept in economics, but it is normally used to understood how equilibrium can develop. Applied to a society, we would expect the declining returns to territorial expansion or administrative complexity (the former often requiring some degree of the latter) to result in eventual stasis rather than collapse. David Friedman has an interesting paper on how the advantages of taxing trade, land or labor gave rise to different equilibria for the sizes & boundaries of polities during the Roman, medieval and nationalist eras in Europe. In Friedman’s theory, each shift between eras resulted from some exogenous change rather being part of the internal logic of societies. Tainter relates some various interesting bits from C. Northcote Parkinson’s “Parkinson’s Law, and Other Studies in Administration”. For example, while “between 1914 and 1967, the number of capital ships in the British Navy declined by 78.9 percent, the number of officers and enlisted men by 32.9 percent, and the number of dockyard workers by 33.7 percent [...] the number of dockyard officials and clerks increased by 247 percent, and the number of Admiralty officals by 767 percent” (emphasis added). Mencius Moldbug would not be surprised to learn that “between 1935 and 1954 the number of officals in the British Colonial office increased by 447 percent” even though “the empire administered by these officials shrank considerably”. These examples are important because they do not demonstrate an increasingly large requirement of administrators for a marginal increase in size/complexity of an entity to be administered, but paying more for less. Parkinson’s explanation was bureaucratic self-serving, which Tainter rejects because he finds trends of increasing hierarchical specialization in the private sector. But because Tainter fails to distinguish between declining marginal returns (eventually reaching zero at a steady-state) and NEGATIVE returns he doesn’t specify whether the latter occurs in the private sector (though Karl Smith would not be surprised if it does for many publicly owned corporations whose shareholders would be better served by liquidation of assets). The growth of administration in higher education would also count, but as a heavily subsidized non-profit sector I can’t say it would qualify. At one point Tainter acknowledges “In many cases this increased, more costly complexity will yield no increased benefits, at other times the benefits will not be proportionate to costs” (emphasis in the original). This is precisely the question at issue of elite mismanagement or the out-of-control inertia of expanding administrative bureaucracies, but as noted he rejected Parkinson’s theory and mocks the idea of societies as runaway trains as self-evidently absurd. Instead he portrays collapse as a choice which is preferable once marginal returns have declined to a certain point. This didn’t entirely make sense to me, because if a society has accidentally shot part the point of zero marginal returns to one of negative returns, the sensible thing is just to reduce that marginal increase in complexity to return to the steady state with zero marginal returns.

The Roman empire sometimes seemed to behave in such a manner, losing some territory and sticking with a more defensible and adminstrable domain (although in Heather’s account some of the lost territory was among the most agriculturally productive), although Tainter thinks the conquests of Britain & Dacia never paid for themselves. So why the path dependency so that changes are not simply reversible? There could be consumption of a not easily renewable resource, a sort of borrowing from the future that leaves future generations deeper in the red. This could happen with soil deterioration, though Tainter doesn’t discuss that much (odd, despite his focus on societies as means of managing sources of energy). His example of Roman emperors increasingly resorting to the debasing of the currency could count (by Diocletian’s time it collected taxes in kind rather than the currency it had rendered nearly worthless), as well as the selling of imperial land. The larger problem in Rome seemed to be an increasingly large portion of subjects who were citizens (both urban proletariat and squabbling elites) subject to fewer or no taxes, while marginal lands were abandoned by overtaxed farmers. An odd feature of the empire was that election officials had to cover the costs of their own office, and as expenses rose there were fewer wealthy people willing to come forward as candidates, until the position was made hereditary. It became obligatory to farm certain deserted lands, with peasants drafted by local city Senates, and Constantine made soldiery a hereditary profession (which required a number of new laws over time to deal with sons who’d rather not follow that career). Taxation of land was simplistic and did not vary based on its quality or yield, so a farmer of marginal land would often be better off working for the owner of a more productive territory and paying rent than failing to cover the taxes on his own plot. With agricultural labor becoming legally tied to the land, we can see the clear beginnings of serfdom and the manorial system. As mentioned, Tainter views the Roman collapse as a choice (as he does others), although of course accounts from the time were more apt to regard it as unfortunate failure or divine punishment.

Interestingly, the “peer polity competition” that replaced Roman civilization is a situation he regards as invulnerable to collapse as opposed to absorbtion, and by removing that “option” he thinks this made peasants demand democratic representation. He acknowledges that this did not happen in the “Warring States” period of China, and instead the Confucian ideology of governance developed. He suggests “Perhaps participatory governance was simply not possible in ancient societies that were so much larger, demographically and territorially, than the Greek city-states”. Someone should have told James Madison (and I’m not being sarcastic). Interestingly enough, there was a civilization of Greek city states which did collapse, just as we’ve mentioned the lowland Maya doing. These are the Mycenaean Greeks who preceded the Dark Ages of Homer’s time. Their collapse is usually attributed to invasion by Dorian Greeks, but Tainter isn’t convinced there’s enough evidence for the Dorians’ presence. Because “Collapse occurs, and can only occur, in a power vacuum” (emphasis in the original) both the Mycenaean Greek and lowland Maya polities must have experienced simultaneous collapse.

The choice of peasants may be limited to passively withdrawing support and just not working very hard (I’d have more to say on that if I’d read James Scott’s “Weapons of the Weak”), but even then I don’t think it’s a desirable outcome for peasants. I’ve mentioned Heather on the greater productivity of Roman territory, and what do you think happened to the masses when that productivity crashed? My understanding of the current archaelogical consensus is that the population crashed as well. Tainter talks about the malnourished skeletons of the peasantry as evidence for the undesirability of certain degrees of complexity, but we also know that English peasants ate better after so many of their peers died of the bubonic plague (it’s also known that peasants have poorer diets than hunter-gatherers, though in Darwinian terms you definitely want to be a farmer). As in James Scott’s account of highland southeast asia (which I don’t entirely buy) was there much cultural defection of the peasantry to the greener pastures outside civilization? Tainter writes that “In 378 [...] Balkan miners went over en masse to the Visigoths”, and that others wished to be conquered/liberated. It is precisely due to the risk of being conquered that he argues is the reason many societies don’t simply revert to a lower level of complexity “even if marginal returns are unfavorable”, but it’s unclear whether conquest is one of outcomes being factored into those marginal returns.

Few people are going to read this book without speculating on their own complex society’s liability to collapse. John Robb and James Kunstler (along with some others in the “Peak Oil” camp) are going to place a high probability on it, while the Singularitarians have the opposite view. Globalization could mean the entire world is now in a state of “peer polity competition” but modern norms (and economic incentives) against conquest and giving war a chance means “failed states” can keep failing for a long time without someone replacing the bad management. Tainter’s studied societies are also Malthusian agricultural ones, it’s hard to know if the same logic will generalize past the industrial revolution. In modern technological economies the costs and benefits of advances may not be simple increasing or declining curves. Robin Hanson doesn’t even consider nearly free energy (which would very important to Tainter) to be nearly as important as the replacement of most human labor by computers (since the latter takes up so much more of GDP). When Tainter was writing there was still just the slightest possibility of nuclear armageddon, now the most likely candidates for death by complexity are grey goo or an unstoppable manmade pandemic. My two cents are that collapse is unlikely in my lifetime, and that’s for the better considering how much worse things could be.

Part of the series AI Risk and Opportunity: A Strategic Analysis.

(You can leave anonymous feedback on posts in this series here. I alone will read the comments, and may use them to improve past and forthcoming posts in this series.)

Building on the previous post on AI risk history, this post provides an incomplete timeline (up to 1993) of significant novel ideas and arguments related to AI as a potential catastrophic risk. I do not include ideas and arguments concerning only, for example, the possibility of AI (Turing 1950) or attempts to predict its arrival (Bostrom 1998).

As is usually the case, we find that when we look closely at a cluster of ideas, it turns out these ideas did not appear all at once in the minds of a Few Great Men. Instead, they grew and mutated and gave birth to new ideas gradually as they passed from mind to mind over the course of many decades.

 

1863: Machine intelligence as an existential risk to humanity; relinquishment of machine technology recommended. Samuel Butler in Darwin among the machines worries that as we build increasingly sophisticated and autonomous machines, they will achieve greater capability than humans and replace humans as the dominant agents on the planet:

...we are ourselves creating our own successors; we are daily adding to the beauty and delicacy of their physical organisation; we are daily giving them greater power and supplying by all sorts of ingenious contrivances that self-regulating, self-acting power which will be to them what intellect has been to the human race. In the course of ages we shall find ourselves the inferior race... the time will come when the machines will hold the real supremacy over the world and its inhabitants...

Our opinion is that war to the death should be instantly proclaimed against them. Every machine of every sort should be destroyed by the well-wisher of his species. Let there be no exceptions made, no quarter shown...

(See also Butler 1872; Campbell 1932.)

1921: Robots as an existential risk. The Czech play R.U.R. by Karel Capek tells the story of robots which grow in power and intelligence and destroy the entire human race (except for a single survivor).

1947: Fragility & complexity of human values (in the context of machine goal systems); perverse instantiation. Jack Williamson's novelette With Folded Hands (1947) tells the story of a race of machines that, in order to follow the Prime Directive: "to serve and obey and guard men from harm." To obey this rule, the machines interfere with every aspect of human life, and humans who resist are lobotomized. Due to the fragility and complexity of human values (Yudkowsky 2008; Muehlhauser and Helm 2012), the machines' rules of behavior had unintended consequences, manifesting a "perverse instantiation" in the language of Bostrom (forthcoming).

(Also see Asimov 1950, 1957, 1983; Versenyi 1974; Minsky 1984; Yudkowsky 2001, 2011.)

1948-1949: Precursor idea to intelligence explosion. Von Neumann (1948) wrote:

...“complication" on its lower levels is probably degenerative, that is, that every automaton that can produce other automata will only be able to produce less complicated ones. There is, however, a certain minimum level where this degenerative characteristic ceases to be universal. At this point automata which can reproduce themselves, or even construct higher entities, become possible.

 

Von Nuemann (1949) came very close to articulating the idea of intelligence explosion:

There is thus this completely decisive property of complexity, that there exists a critical size below which the process of synthesis is degenerative, but above which the phenomenon of synthesis, if properly arranged, can become explosive, in other words, where syntheses of automata can proceed in such a manner that each automaton will produce other automata which are more complex and of higher potentialities than itself.

1951: Potentially rapid transition from machine intelligence to machine takeover. Turing (1951) described ways that intelligent computers might learn and improve their capabilities, concluding that:

...it seems probable that once the machine thinking method has started, it would not take long to outstrip our feeble powers... At some stage therefore we should have to expect the machines to take control...

1959: Intelligence explosion; the need for human-friendly goals for machine superintelligence. Good (1959) describes what he later (1965) called an "intelligence explosion," a particular mechanism for rapid transition from artificial general intelligence to dangerous machine takeover:

Once a machine is designed that is good enough… it can be put to work designing an even better machine. At this point an "explosion" will clearly occur; all the problems of science and technology will be handed over to machines and it will no longer be necessary for people to work. Whether this will lead to a Utopia or to the extermination of the human race will depend on how the problem is handled by the machines. The important thing will be to give them the aim of serving human beings.

(Also see Good 1965, 1970; Vinge 1992, 1993; Yudkowsky 2008.)

1966: A military arms race for machine superintelligence could accelerate machine takeover; convergence toward a singleton is likely. Dennis Feltham Jones' 1966 novel Colossus depicted what may be a particularly likely scenario: two world superpowers (the USA and USSR) are in an arms race to develop superintelligent computers, one of which self-improves enough to take control of the planet.

In the same year, Cade (1966) argued the same thing:

 

political leaders on Earth will slowly come to realize... that intelligent machines having superhuman thinking ability can be built. The construction of such machines, even taking into account all the latest developments in computer technology, would call for a major national effort. It is only to be expected that any nation which did put forth the financial and physical effort needed to build and programme such a machine, would also attempt to utilize it to its maximum capacity, which implies that it would be used to make major decisions of national policy. Here is where the awful dilemma arises. Any restriction to the range of data supplied to the machine would limit its ability to make effective political and economic decisions, yet if no such restrictions are placed upon the machine's command of information, then the entire control of the nation would virtually be surrendered to the judgment of the robot.

On the other hand, any major nation which was led by a superior, unemotional intelligence of any kind, would quickly rise to a position of world domination. This by itself is sufficient to guarantee that, sooner or later, the effort to build such an intelligence will be made — if not in the Western world, then elsewhere, where people are more accustomed to iron dictatorships.

...It seems that, in the forseeable future, the major nations of the world will have to face the alternative of surrendering national control to mechanical ministers, or being dominated by other nations which have already done this. Such a process will eventually lead to the domination of the whole Earth by a dictatorship of an unparalleled type — a single supreme central authority.

 

(This last paragraph also argues for convergence toward what Bostrom later called a "singleton.")

(Also see Ellison 1967.)

1970: Proposal for an association that analyzes the implications of machine superintelligence; naive control solutions like "switch off the power" may not work because the superintelligence will outsmart us, thus we must focus on its motivations; possibility of "pointless" optimization by machine superintelligence. Good (1970) argues:

Even if the chance that the ultraintelligent machine will be available [soon] is small, the repercussions would be so enormous, good or bad, that it is not too early to entertain the possibility. In any case by 1980 I hope that the implications and the safeguards will have been thoroughly discussed, and this is my main reason for airing the matter: an association for considering it should be started.

(Also see Bostrom 1997.)

On the idea that naive control solutions like "switch off the power" may not work because the superintelligence will find a way to outsmart us, and thus we must focus our efforts on the superintelligence's motivations, Good writes:

Some people have suggested that in order to prevent the [ultraintelligent machine] from taking over we should be ready to switch of its power supply. But it is not as simple as that because the machine could recommend the appointment of its own operators, it could recommend that they be paid well and it could select older men who would not be worried about losing their jobs. Then it could replace its operators by robots in order to make sure that it is not switched off. Next it could have the neo-Luddites ridiculed by calling them Ludditeniks, and if necessary it would later have them imprisoned or executed. This shows how careful we must be to keep our eye on the "motivation" of the machines, if possible, just as we should with politicians.

(Also see Yudkowsky 2008.)

Good also outlines one possibility for "pointless" goal-optimization by machine superintelligence:

If the machines took over and men became redundant and ultimately extinct, the society of machines would continue in a complex and interesting manner, but it would all apparently be pointless because there would be no one there to be interested. If machines cannot be conscious there would be only a zombie world. This would perhaps not be as bad as in many human societies where most people have lived in misery and degradation while a few have lived in pomp and luxury. It seems to me that the utility of such societies has been negative (while in the condition described) whereas the utility of a zombie society would be zero and hence preferable.

(Also see Bostrom 2004; Yudkowsky 2008.)

1974: We can't much predict what will happen after the creation of machine superintelligence. Julius Lukasiewicz (1974) writes:

The survival of man may depend on the early construction of an ultraintelligent machine-or the ultraintelligent machine may take over and render the human race redundant or develop another form of life. The prospect that a merely intelligent man could ever attempt to predict the impact of an ultraintelligent device is of course unlikely but the temptation to speculate seems irresistible.

(Also see Vinge 1993.)

1977: Self-improving AI could stealthily take over the internet; convergent instrumental goals in AI; the treacherous turn. Though the concept of a self-propagating computer worm was introduced by John Brunner's The Shockwave Rider (1975), Thomas J. Ryan's novel The Adolescence of P-1 (1977) tells the story of an intelligent worm that at first is merely able to learn to hack novel computer systems and use them to propagate itself, but later (1) has novel insights on how to improve its own intelligence, (2) develops convergent instrumental subgoals (see Bostrom 2012) for self-preservation and resource acquisition, and (3) learns the ability to fake its own death so that it can grow its powers in secret and later engage in a "treacherous turn" (see Bostrom forthcoming) against humans.

1982: To design ethical machine superintelligence, we may need to design superintelligence first and then ask it to solve philosophical problems (e.g. including ethics).

Good (1982) writes:

Unfortunately, after 2500 years, the philosophical problems are nowhere near solution. Do we need to solve these philosophical problems before we can design an adequate ethical machine, or is there another approach? One approach that cannot be ruled out is first to produce an ultra-intelligent machine and then ask it to solve philosophical problems.

1988: Even though AI poses an existential threat, we may need to rush toward it so we can use it to mitigate other existential threats. Moravec (1988, p. 100-101) writes:

...intelligent machines... threaten our existence... Machines merely as clever as human beings will have enormous advantages in competitive situations... So why rush headlong into an era of intelligent machines? The answer, I believe, is that we have very little choice, if our culture is to remain viable... The universe is one random event after another. Sooner or later an unstoppable virus deadly to humans will evolve, or a major asteroid will collide with the earth, or the sun will expand, or we will be invaded from the stars, or a black hole will swallow the galaxy. The bigger, more diverse, and competent a culture is, the better it can detect and deal with external dangers. The larger events happen less frequently. By growing rapidly enough, a culture has a finite chance of surviving forever.

1993: Physical confinement is unlikely to constrain superintelligences, for superintelligences will outsmart us. Vinge (1993) writes:

I argue that confinement [of superintelligent machines] is intrinsically impractical. For the case of physical confinement: Imagine yourself confined to your house with only limited data access to the outside, to your masters. If those masters thought at a rate — say — one million times slower than you, there is little doubt that over a period of years (your time) you could come up with "helpful advice" that would incidentally set you free...

After 1993. The extropians mailing list was launched in 1991, and was home to hundreds of discussions in which many important new ideas were proposed — ideas later developed in the public writings of Bostrom, Yudkowsky, Goertzel, and others. Unfortunately, the discussions from before 1998 were private, by agreement among subscribers. The early years of the archive cannot be made public without getting permission from everyone involved — a nearly impossible task. I have, however, collected all posts I could find from 1998 onward and uploaded them here (link fixed 04-03-2012).

I will end this post here. Perhaps in a future post I will extend the timeline past 1993, when interest in the subject became greater and thus the number of new ideas generated per decade rapidly increased.

References

• Asimov (1950). The Evitable Conflict
• Asimov (1957). The Naked Sun
• Asimov (1983). The Robots of Dawn
• Bostrom (1997). Predictions from Philosophy? How philosophers could make themselves useful
• Bostrom (1998). How Long before Superintelligence?
• Bostrom (2004). The Future of Human Evolution
• Bostrom (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents
• Bostrom (forthcoming). Superintelligence.
• Brunner (1975). The Shockwave Rider
• Butler (1863). Darwin among the machines
• Butler (1872). Erewhon.
• Campbell (1932). The Last Evolution
• Capek (1921). R.U.R.
• Ellison (1967). I Have No Mouth, and I Must Scream
• Good (1959). Speculations on perceptrons and other automata
• Good (1965). Speculations Concerning the First Ultraintelligent Machine
• Good (1970). Some future social repercussions of computers
• Jones (1966). Colossus.
• Lukasiewicz (1974). The Ignorance Explosion.
• Minsky (1984). Afterward to Vinge's 'True Names'.
• Moravec (1988). Mind Children: The Future of Robot and Human Intelligence.
• Muehlhauser & Helm (2012). The Singularity and Machine Ethics
• Ryan (1977). The Adolescence of P-1
• Turing (1950). Computing Machinery and Intelligence
• Turing (1951). Intelligent machinery, a heretical theory
• Versenyi (1974). Can robots be moral?
• Vinge (1992). A Fire Upon The Deep.
• Vinge (1993). The Coming Technological Singularity.
• Von Neumann (1948). The general and logical theory of automata.
• Von Neumann (1949). Theory and Organization of Complicated Automata. (Five lectures delivered at the University of Illinois in December, 1949. Reprinted in Papers of John Von Neumann on Computers and Computing Theory.)
• Williamson (1947). With Folded Hands.
• Yudkowsky (2001). Creating Friendly AI.
• Yudkowsky (2008). Artificial Intelligence as a Positive and Negative Factor in Global Risk
• Yudkowsky (2011). Complex value systems are required to realize valuable futures

Yay a new cool post is up on West Hunters blog! It is written by Gregory Cochran and Henry Harpending with whom most LWers are probably already familiar with (particularly this awesome entry). It raises some interesting points on biases in academia.

I was contemplating Conan the Barbarian, and remembered the essay that Robert E. Howard wrote about the  background of those stories – The Hyborian Age.  I think that the flavor of Howard’s pseudo-history is a lot more realistic than the picture of the human past academics preferred over the past few decades.

In Conan’s world, it’s never surprising to find a people that once mixed with some ancient prehuman race.  Happens all the time.  Until very recently, the vast majority of workers in human genetics and paleontology were sure that this never occurred – and only changed their minds when presented with evidence that was both strong (ancient DNA)  and too mathematically sophisticated for them to understand or challenge (D-statistics).

Conan’s history  was shaped by the occasional catastrophe.  Most academics (particularly geologists) don’t like catastrophes, but they have grudgingly come to admit their importance – things like the Thera and Toba eruptions, or the K/T asteroid strike and the Permo-Triassic crisis.

Between the time when the oceans drank Atlantis, and the rise of the sons of Aryas, evolution seems to have run pretty briskly, but without any pronounced direction.  Men devolved into ape-men when the environment pushed in that direction (Flores ?)  and shifted right back when the environment favored speech and tools.  Culture shaped evolution, and evolution shaped culture.  An endogamous caste of snake-worshiping priests evolved in a strange direction.  Although their IQs were considerably higher than average, they remained surprisingly vulnerable to sword-bearing barbarians.

In this world, evolution could happen on a time scale of thousands of years, and there was no magic rule that ensured that the outcome would be the same in every group.  It may not be PC to say it, but Cimmerians were smarter than Picts.

The basic idea of their book "The 10 000 Year Explosion" (LessWrong review, Amazon).

Above all, people in Conan’s world fought. They migrated: they invaded.  There was war before, during, and after civilization.  Völkerwanderungs were a dime a dozen. Conquerors spread.  Sometimes they mixed with the locals, sometimes they replaced them – as when the once dominant Hyborians, overrun by Picts, vanished from the earth, leaving scarcely a trace of their blood in the veins of their conquerors. They must have been U5b.

To be fair,  real physical anthropologists in Howard’s day thought that there had been significant population movements and replacements in Europe, judging from changes in skeletons and skulls that accompanied archeological shifts, as when people turned taller, heavier boned , and brachycephalic just as the Bell-Beaker artifacts show up. But those physical anthropologists lost out to people like Boas – liars.

Perhaps this little old entry is relevant here. ^_^

Given the chance (sufficient lack of information), American anthropologists assumed that the Mayans were peaceful astronomers. Howard would have assumed that they were just another blood-drenched snake cult: who came closer?

Now I’m not saying that Howard got every single tiny little syllable of prehistory right.  Not likely: so far, we haven’t seen any signs of Cthulhu-like visitors, which abound in the Conan stories.  So far. But Howard’s priors were more accurate than those of the pots-not-people archeologists: more accurate than people like Excoffier and  Currat, who assume that there hasn’t been any population replacement in Europe since moderns displaced Neanderthals. More accurate than Chris Stringer,  more accurate than Brian Ferguson.

Most important, Conan, unlike the typical professor, knew what was best in life.

Heh.

Cochran you are such a nerd.

What would the world look like without Hitler? Fiction is generally unequivocal about this: the removal of Hitler makes no difference, the world will still lurch towards a world war through some other path. WWII and the Holocaust are such major, defining events of the twentieth century, that we twist counterfactual events to ensure they still happen.

Against this, some have made the argument that Hitler was essentially sole responsible for WWII and especially for the Holocaust - no Hitler, no war, no extermination camps. The no Holocaust argument is quite solid: the extermination system was expensive, militarily counter-productive, and could only have happened given a leader lacking checks and balance and with an idée fixe that overrode everything else (general European antisemitism allowed the Holocaust, but didn't cause it). The no WWII argument points out that Hitler was both irrational and lucky: he often took great risks, on flimsy evidence, and got away with them. Certainly his decisions in the later, post-Barbarossa period of his reign belie political, military or organisational genius. And it was the height of stupidity to have gone to war, for a half of Poland, with simultaneously the world's greatest empire and what appeared to be the overwhelmingly strong French army. Yes Gamelin, the French commander in chief, did behave like a concussed duckling, and the German army outfought the French - but no-one could have predicted this, and no-one sensible would have counted on it, and hence they wouldn't have risked the war. Hitler wan't sensible, and lucked out.

I am an aspiring historian and I'm very interested in ways to apply Bayesian reasoning to history. When I say "history" I mean the study of history -- as a historian, allowing my map of what has happened in the past to match the territory, and being able to represent more accurately the relative strength of historical evidence for and against various historical models.

I know that historical evidence works quite a bit differently from scientific evidence. But I think that historical evidence is also useful. Historians, in recording and assembling secondary sources, assess the relative strength of evidence (mostly primary sources) with regards to a topic already.  But there must be a way to do it more formally. Shouldn't there be a right answer, just as no two people who are completely rational (and have the same information) should ever disagree?

This is a post (or series of posts) I might write in the future, and I have put a bit of thought into it so far, but I need to do quite a bit more research. Is there anyone interested in reading something on this topic? Has it been done before? Is there anyone who is knowledgeable about how historians treat evidence who might be able to offer some insights?

Background: Apostles' Creed, Tsuyoku Naritai

Related to: A Parable on Obsolete Ideologies

Just something I thought I might add to the annals of cases where someone tries to re-interpret an old religious text to mean something more acceptable to the modern ear, in contradiction to what most people (especially its contemporaries) think the texts mean.  And this is not some random person, but Gene Callahan, who makes sure you understand he holds a doctorate in philosophy, and pretty much makes a career out of defending this and anti-reductionist views in general.  Here's the post:

I believe in one God, the Father Almighty, Maker of heaven and earth...
And in one Lord Jesus Christ, the only-begotten Son of God...
And I believe in the Holy Ghost, the Lord and Giver of Life..."

If I say these words, what do I mean? I am asserting that I have some secret knowledge that others do not? Do I believe these things like I believe it will rain tonight?

No, I asserting that, by meditating on these symbols, I believe I will come to understand better what I now know only through a glass darkly.

I believe that I may understand.

I suggested that this is not what most people mean when they say the Creed, but (surprise) the comment was deleted.

(Yes I know Tsuyoku Naritai is not quite the same as Callahan's interpretation, but it's the closest short LW term for the general idea.)

There was a historical shift in beliefs.

But the one we’re concerned with is that women’s libidos went from being considered as powerful or more so than men’s to being essentially erased. Pre-Renaissance examples of horny ladies abound, from the Greeks onward; make your own list, but do include Chaucer. He’s such fun. This change in attitudes appears to have been religiously motivated, and based on the idea that women are more spiritual and sacred than men, meaning less horny. Again, make your own list of contemporary leftovers of this attitude; there are plenty.

By the 18th century, it was taken as read that a woman who did experience (or at least express) sexual desire was suffering from a disorder. One important 1775 study of the subject linked the problem to “secret pollutions”, i.e. wanking, and (I swear I am not making this up) eating too much chocolate. I guess that’d go a ways toward explaining this advertisement. Women were diagnosed with, treated for, and often operated upon for “nymphomania”, the dread condition that causes a woman to want sex. (Talk to your doctor; you may suffer from it yourself!) And yes, by “operated upon”, I mean clitoridectomy. And yes, that’s fucking appalling.

I find this very odd. How could a major cultural lineage be wrong about something so much a part of ordinary experience?

When I say wrong, I don't necessarily mean that we're right, or the ancients were right, though there's a lot of evidence that the Victorians were wrong.

My favorite theory is that people's amount of desire for sex varies sufficiently that there's enough noise to make it easy to see patterns that aren't there. I leave the possibility open that there was a change (possibly dietary) which affected libido levels differently between men and women.

People are sufficiently punitive about sex that there's going to be lies and misdirection to support the current theory about how people are supposed to be.

I think LWers may be intrigued...

Tim McGrew, author of this excellent annotated bibliography on Bayesian reasoning, recently co-authored with his wife Lydia a Bayesian defense of the resurrection of Jesus. I interviewed Lydia for my podcast, here. Atheist Richard Carrier has leveled some objections to their article, but his objections are weak.

Have at it.

I'm reading a popular science encyclopedia now, particularly chapters about the history of physics. The chapter goes on to evaluate the development of the concept of kinetic energy, starting with Aristotle's (grossly incorrect) explanation of a flying arrow saying that it's kept in motion by the air behind it, and then continuing to medieval impetus theory. Added: The picture below illustrates the trajectory of a flying cannonball as described by Albert of Saxony.

What struck me immediately was how drastically different from observations its predictions were. The earliest impetus theory predicted that a cannonball's trajectory was an angle: first a slanted straight line until the impetus runs out, then a vertical line of freefall. A later development added an intermediate stage, as seen on the picture to the left. At first the impetus was at full force, and would launch the cannonball in a straight line; then it would gradually give way to freefall and curve until the ball would be falling in a straight line.

While this model is closer to reality than the original prediction, I still cannot help but think... How could they deviate from observations so strongly?

Yes, yes, hindsight bias.

But if you launch a stream of water out of a slanted tube or sleeve, even if you know nothing about paraboles, you can observe that the curve it follows in the air is symmetrical. Balls such as those used for games would visibly not produce curves like depicted.

Perhaps the idea of verifying theories with experiments was only beginning to coalesce at that time, but what kind of possible thought process could lead one to publish theories so grossly out of touch with everyday observations, even those that you see without making any explicit experiments? Did the authors think something along the lines of "Well, reality should behave this way, and if it doesn't, it's its own fault"?

>Elisabeth Lloyd: I don’t actually know. I think that it’s at a very problematic intersection of topics. I mean, you’re taking the intersection of human evolution, women, sexuality – once you take that intersection you’re bound to kind of get a disaster. More than that, when evolutionists have looked at this topic, I think that they’ve had quite a few items on their agenda, including telling the story about human origins that bolsters up the family, monogamy, a certain view of female sexuality that’s complimentary to a certain view of male sexuality. And all of those items have been on their agenda and it’s quite visible in their explanations.

>Natasha Mitchell: I guess it’s perplexed people partly, too, because women don’t need an orgasm to become pregnant, and so the question is: well, what’s its purpose? Well, is its purpose to give us pleasure so that we have sex, so that we can become pregnant, according to the classic evolutionary theories?

>Elisabeth Lloyd: The problem is even worse than it appears at first because not only is orgasm not necessary on the female side to become pregnant, there isn’t even any evidence that orgasm makes any difference at all to fertility, or pregnancy rate, or reproductive success. It seems intuitive that a female orgasm would motivate females to engage in intercourse which would naturally lead to more pregnancies or help with bonding or something like that, but the evidence simply doesn’t back that up.

The whole discussion. It backs my theory that using evolution to explain current traits seriously tempts people to make things up.