In partially observable environments, stochastic policies can be optimal
I always had the informal impression that the optimal policies were deterministic (choosing the best option, rather than some mix of options). Of course, this is not the case when facing other agents, but I had the impression this would hold when facing the environment rather that other players.
But stochastic policies can also be needed if the environment is partially observable, at least if the policy is Markov (memoryless). Consider the following POMDP (partially observable Markov decision process):

There are two states, 1a and 1b, and the agent cannot tell which one they're in. Action A in state 1a and B in state 1b, gives a reward of -R and keeps the agent in the same place. Action B in state 1a and A in state 1b, gives a reward of R and moves the agent to the other state.
The returns for the two deterministic policies - A and B - are -R every turn except maybe for the first. While the return for the stochastic policy of 0.5A + 0.5B is 0 per turn.
Of course, if the agent can observe the reward, the environment is no longer partially observable (though we can imagine the reward is delayed until later). And the general policy of "alternate A and B" is more effective that the 0.5A + 0.5B policy. Still, that stochastic policy is the best of the memoryless policies available in this POMDP.
Paid research assistant position focusing on artificial intelligence and existential risk
Yale Assistant Professor of Political Science Allan Dafoe is seeking Research Assistants for a project on the political dimensions of the existential risks posed by advanced artificial intelligence. The project will involve exploring issues related to grand strategy and international politics, reviewing possibilities for social scientific research in this area, and institution building. Familiarity with international relations, existential risk, Effective Altruism, and/or artificial intelligence are a plus but not necessary. The project is done in collaboration with the Future of Humanity Institute, located in the Faculty of Philosophy at the University of Oxford. There are additional career opportunities in this area, including in the coming academic year and in the future at Yale, Oxford, and elsewhere. If interested in the position, please email allan.dafoe@yale.edu with a copy of your CV, a writing sample, an unofficial copy of your transcript, and a short (200-500 word) statement of interest. Work can be done remotely, though being located in New Haven, CT or Oxford, UK is a plus.
The Art of Lawfare and Litigation strategy
Bertrand Russell, well aware there were health risks of smoking, defended his addiction in a videotaped interview. See if you can spot his fallacy!
Today on SBS (radio channel in Australia) I heard reporters breaking the news that Nature article reports that Cancer is largely due to choices. I was shocked by what appeared to be gross violations of cultural norms around the blaming of victims. I wanted to investigate further since science reporting is notoriously inaccurate.
The BBC reports:
Earlier this year, researchers sparked a debate after suggesting two-thirds of cancer types were down to luck rather than factors such as smoking.
The new study, in the journal Nature, used four approaches to conclude only 10-30% of cancers were down to the way the body naturally functions or "luck".
"They can't smoke and say it's bad luck if they have cancer."
-Dr Yusuf Hannun, the director of Stony Brook
The BBC article is roughly concordant with the SBS report.
I've had a fairly simple relationship with cigarettes. I've smoked others' cigarettes a few times, while drinking. I bought my first cigarette to try soon after I turned of age and discarded the rest of the packet. One of my favourite memories is trying a vanilla flavoured cigar. I still feel tempted to it again whenever I smell a nice scent, or think about that moment. Though now, I regularly reject offers to go to local venues and smoke hookah. Even after my first cigarette, I felt the tug of nicotine and tobacco. Though, I'm unusually sensitive to eve the mildest addictive substances, so that doesn't suprise me in respective. What does suprise me, is that society is starting to take a ubiquitous but increasingly undeniable health issue seriously despite deep entanglement with long standing way of doing things, political ideologues, individual addictions and addiction-driven political behaviour and shareholder's pockets.
Though the truth claim of the article isn't that suprising. The dangers of smoking are publicised everywhere. Emphasis mine:
13 die every day in Victoria as a result of smoking.
Tobacco use (which includes cigarettes, cigars, pipes, snuff, chewing tobacco) is the leading preventable cause of death and illness in our country. It causes more deaths annually than those killed by AIDS, alcohol, automobile accidents, murders, suicides, drugs and fires combined.
So I decided to learn more about the relationship between society and big tobacco, and government and big tobacco to see what other people interested in influencing public policy and public health can learn (effective altruism policy analytics, take note!) about policy tractability in suprising places.
Here's what might make for tractable public policy for public health interventions
Proof of concept
Governments are great at successfully suing the shit out of tobacco. And, big tobacco takes it like a champ:
It started with United State's states experimenting with suing big tobacco. Eventually only a couple of states hadn't done it. Big Tobacco and all those attorney generals gathered and arranged huge ass settlement that resulted in the disestablishment of several shill research institutes supporting big tobacco and big payouts to sponsor anti-smoking advocacy groups (which seem politically unethical, but consequentially good, but I suppose that's a different story). However, what's important to note here is the experimentation within US states culminating with the legitimacy of normative lawfare. It's called 'Diffusion theory' and is described here.
Wait wait wait. I know what you're thinking, non-US LessWrongers - another US centric analysis that isn't too transportable. No. I'm not American in any sense, it's just that the US seems to be a point of diffusion. What's happening regarding marajuana in the US now seems to mirror this in some sense, but it's ironically pro-smoking. That illustrates the cause-neutrality of this phenomenon.
That settlement wasn't the end of the lawfare:
On August 17, 2006, a U.S. district judge issued a landmark opinion in the government's case against Big Tobacco, finding that tobacco companies had violated civil racketeering laws and defrauded consumers by lying about the health risks of smoking.
In a 1,653 page ruling, the judge stated that the tobacco industry had deceived the American public by concealing the addictive nature of nicotine plus had targeted youth in order to get them hooked on cigarettes for life. (Appeals are still pending).
Victims who ask for help
I also stumbled upon some smokers attitudes to smoking and their, well, seemingly vexacious attitudes to big tobacco when looking up lawsuits and big tobacco. Here's a copy of the comments section on one website. It's really heartbreaking. It's a small sample size but just note their education too - suggesting a socio-economic effect. Note, this comments were posted publicly and are blatant cries for help. This suggests political will at a grassroots level that is yet under-catered for by services and/or political action. That's a powerful thing, perhaps - visible need in public forums addressed to those that are in the relevant space. Note that they commented on a class action website.
http://s10.postimg.org/61h7b1rp5/099090.png

Note some of the language:
"I feel like I'm being tortured"
You don't see that kind of language used in any effective altruism branded publications.
Villains
Somewhat famous documents exposing the tobacco industries internal motivations and dodginess seem to be quoted everywhere in websites documenting and justifyications of lawfare against the tobacco industry. Public health and personal dangers of smoking don't seem to have been the big catalyst, but rather a villainous enemy. I'm reminded of how the Stop the boats campaign which villainised people smugglers instead of speaking of the potential to save lives of refugees who fall overboard shitty vessals. I think to Open Borders campaigners associated with GiveWell's Open Philanthropy Project, the perception of the project as just about the most intractable policy prospect around (I'd say a moratorium on AI research is up there), but at the same time, non identification of a villain in the picture. That's not entirely unsuprising. I recall the hate I received when I suggested that people should consider prostituting themselves for effective altruism, or soliciting donations from the porn industry where donors struggle to donate since many, particularly relgious charities refuge to accept their donations. Likewise, it's hard to get rid of encultured perceptions of what's good and what's bad, rather then enumerating ('or checking, as Eleizer writes in the sequence) the consequences.
Relative merit
This is something Effective Altruist is doing.
William Savedoff and Albert Alwang recently identified taxes on tobacco as, “the single most cost-effective way to save lives in developing countries” (2015, p.1).
...
Tobacco control programs often pursue many of these aims at once. However, raising taxes appears to be particularly cost-effective — e.g., raising taxes costs $3 - $70 per DALY avoided(Savedoff and Alwang, p.5; Ranson et al. 2002, p.311) — so I will focus solely on taxes. I will also focus only on low and middle income countries (LMICs) because that is where the problem is worst and where taxes can do the most good most cost-effectively.
..
But current trends need not continue. We can prevent deaths from tobacco use. Tobacco taxation is a well-tested and effective means of decreasing the prevalence of smoking—it gets people to stop and prevents others from starting. The reason is that smokers are responsive to price increases,provided that the real price goes up enough
...
Even if these numbers are off by a factor of 2 or 3, tobacco taxation appears to be on par with the most effective interventions identified by GiveWell and Giving What We Can. For example, GiveWell estimates that AMF can prevent a death for $3340 by providing bed nets to prevent malaria and estimates the cost of schistosomiasis deworming at $29 - $71 per DALY.
There are a few reasons to balk at recommending tobacco tax advocacy to those aiming to do the most good with their donations, time, and careers.
- Tobacco taxes may not be a tractable issue
- Tobacco taxes may be a “crowded” cause area
- Unanswered questions about the empirical basis of cost-effectiveness estimates
...
- There may not be a charity to donate to
Smoking is very harmful and very common. Globally, 21% of people over 15 smoke (WHO GHO)
-https://www.givingwhatwecan.org/post/2015/09/tobacco-control-best-buy-developing-world/
Attributing public responsibility AND incentivising independently private interest in a cause
The Single Best Health Policy in the World: Tobacco Taxes
The single most cost-effective way to save lives in developing countries is in the hands of developing countries themselves: raising tobacco taxes. In fact, raising tobacco taxes is better than cost-effective. It saves lives while increasing revenues and saving poor households money when their members quit smoking.
-http://www.cgdev.org/publication/single-best-health-policy-world-tobacco-taxes)
Tobacco lawsuits can be hard to win but if you have been injured because of tobacco or smoking or secondary smoke exposure, you should contact an attorney as soon as possible.
If you have lung cancer and are now, or were formerly, a smoker or used tobacco products, you may have a claim under the product liability laws. You should contact an experienced product liability attorney or a tobacco lawsuit attorney as soon as possible because a statute of limitations could apply.
There's a whole bunch of legal literature like this: http://heinonline.org/HOL/LandingPage?handle=hein.journals/clqv86&div=45&id=&page=
that I don't have the background to search for and interpret. So, if I'm missing important things, perhaps it's attributable to that. Point them out please.
So that's my analysis: plausible modifiable variables that influence the tractability of the public health policy initiative:
(1) Attributing public responsibility AND incentivising independently private interest in a cause
(2) Relative merit
(3) Villains
(4) Victims that ask for help
(5) Low scale proof of concept
Remember, lawfare isn't just the domain of governments. Here's an example of non-government lawfare for public health. They are just better resourced, often, than individuals. They need groups to advocate on their behalf. Perhaps that's a direction the Open Philanthropy Project could take.
I want to finish by soliciting an answer on the following question that is posed to smokers in a recurring survey by a tobacco control body:
Do you support or oppose the government suing tobacco companies to recover health care costs caused by tobacco use?
Now, there may be some 'reverse causation' at play here for why Tobacco Control has been so politically effect. BECAUSE it's such a good cause, it's a low hanging fruit that's already being picked.
What's the case for or against this?
The case for it's cause selection: Tobacco control
Importance: high
tobacco is the leading preventable cause of death and disease in both the world (see: http://www.who.int/nmh/publications/fact_sheet_tobacco_en.pdf) and Australia (see: http://www.cancer.org.au/policy-and-advocacy/position-statements/smoking-and-tobacco-control/)
‘Tobacco smoking causes 20% of cancer deaths in Australia, making it the highest individual cancer risk factor. Smoking is a known cause of 16 different cancer types and is the main cause of Australia’s deadliest cancer, lung cancer. Smoking is responsible for 88% of lung cancer deaths in men and 75% of lung cancer cases in women in Australia.’
Tractable: high
The World Health Organization’s Framework Convention on Tobacco Control (FCTC) was the first public health treaty ever negotiate.
Based on private information, the balance of healthcare costs against tax revenues according to health advocates compared to treasury estimates in Australia may have been relevant to Australia’s leadership in tobacco regulation. That submission may or may not be adequate in complexity (ie. taking into account reduced lifespans impact on reduced pension payouts for instance). There is a good article about the behavioural economics of tobacco regulation here (http://baselinescenario.com/2011/03/22/incentives-dont-work/)
Room for advocacy: low
There are many hundreds of consumer support and advocacy groups, and cancer charities across Australia.
Room for employment: low?
Room for consulting: high
The rigour of analysis and achievements themselves in the Cancer Council of Australia annual review is underwhelming, as is the Cancer Council of Victoria’s annual report. There is a better organised body of evidence relating to their impact on their Wiki pages about effective interventions and policy priorities. At a glance, there appears to be room for more quantitative, methodologically rigorous and independent evaluation. I will be looking at GiveWell to see what I recommendations can be translated. I will keep records of my findings to formulate draft guidelines for advising organisations in the Cancer Councils’ positions which I estimate by vague memory of GiveWell’s claims are in the majority in the philanthropic space.
AI: requirements for pernicious policies
Some have argued that "tool AIs" are safe(r). Recently, Eric Drexler decomposed AIs into "problem solvers" (eg calculators), "advisors" (eg GPS route planners), and actors (autonomous agents). Both solvers and advisors can be seen as examples of tools.
People have argued that tool AIs are not safe. It's hard to imagine a calculator going berserk, no matter what its algorithm is, but it's not too hard to come up with clear examples of dangerous tools. This suggests the solvers vs advisors vs actors (or tools vs agents, or oracles vs agents) is not the right distinction.
Instead, I've been asking: how likely is the algorithm to implement a pernicious policy? If we model the AI as having an objective function (or utility function) and algorithm that implements it, a pernicious policy is one that scores high in the objective function but is not at all what is intended. A pernicious function could be harmless and entertaining or much more severe.
I will lay aside, for the moment, the issue of badly programmed algorithms (possibly containing its own objective sub-functions). In any case, to implement a pernicious function, we have to ask these questions about the algorithm:
- Do pernicious policies exist? Are there many?
- Can the AI find them?
- Can the AI test them?
- Would the AI choose to implement them?
The answer to 1. seems to be trivially yes. Even a calculator could, in theory, output a series of messages that socially hack us, blah, take over the world, blah, extinction, blah, calculator finishes its calculations. What is much more interesting is some types of agents have many more pernicious policies than others. This seems the big difference between actors and other designs. An actor AI in complete control of the USA or Russia's nuclear arsenal has all sort of pernicious policies easily to hand; an advisor or oracle has much fewer (generally going through social engineering), a tool typically even less. A lot of the physical protection measures are about reducing the number of sucessfull pernicious policies the AI has a cess to.
The answer to 2. is mainly a function of the power of the algorithm. A basic calculator will never find anything dangerous: its programming is simple and tight. But compare an agent with the same objective function and the ability to do an unrestricted policy search with vast resources... So it seems that the answer to 2. does not depend on any solver vs actor division, but purely on the algorithm used.
And now we come to the big question 3., whether the AI can test these policies. Even if the AI can find pernicious policies that rank high on its objective function, it will never implement them unless it can ascertain this fact. And there are several ways it could do so. Let's assume that a solver AI has a very complicated objective function - one that encodes many relevant facts about the real world. Now, the AI may not "care" about the real world, but it has a virtual version of that, in which it can virtually test all of its policies. With a detailed enough computing power, it can establish whether the pernicious policy would be effective at achieving its virtual goal. If this is a good approximation of how the pernicious policy would behave in the real world, we could have a problem.
But extremely detailed objective functions are unlikely. But even simple ones can show odd behaviour if the agents gets to interact repeatedly with the real world - this is the issue with reinforcement learning. Suppose that the agent attempts a translation job, and is rewarded on the accuracy of its translation. Depending on the details of what the AI knows and who choose the rewards, the AI could end up manipulating its controllers, similarly to this example. The problem is that one there is any interaction, all the complexity of humanity could potentially show up in the reward function, even if the objective function is simple.
Of course, some designs make this very unlikely - resetting the AI periodically can help to alleviate the problem, as can choosing more objective criteria for any rewards. Lastly on this point, we should mention the possibility that human R&D, by selecting and refining the objective function and the algorithm, could take the roll of testing the policies. This is likely to emerge only in cases where many AI designs are considered, and the best candiates are retained based on human judgement.
Finally we come to the question of whether the AI will implement the policy if it's found it and tested it. You could say that the point of FAI is to create an AI that doesn't choose to implement pernicious policies - but, more correctly, the point of FAI is to ensure that very few (or zero) pernicious policies exist in the first place, as they all score low on the utility function. However, there are a variety of more complicated designs - satisficers, agents using crude measures - where the questions of "Do pernicious policies exist?" and "Would the AI choose to implement them?" could become quite distinct.
Conclusion: a more through analysis of AI designs is needed
A calculator is safe, because it is a solver, it has a very simple objective function, with no holes in the algorithm, and it can neither find nor test any pernicious policies. It is the combination of these elements that makes it almost certainly safe. If we want to make the same claim about other designs, neither "it's just a solver" or "it's objective function is simple" would be enough; we need a careful analysis.
Though, as usual, "it's not certainly safe" is a quite distinct claim from "it's (likely) dangerous", and they should not be conflated.
How well will policy-makers handle AGI? (initial findings)
Cross-posted from MIRI's blog.
MIRI's mission is "to ensure that the creation of smarter-than-human intelligence has a positive impact." One policy-relevant question is: How well should we expect policy makers to handle the invention of AGI, and what does this imply about how much effort to put into AGI risk mitigation vs. other concerns?
To investigate these questions, we asked Jonah Sinick to examine how well policy-makers handled past events analogous in some ways to the future invention of AGI, and summarize his findings. We pre-committed to publishing our entire email exchange on the topic (with minor editing), just as with our project on how well we can plan for future decades. The post below is a summary of findings from our full email exchange (.docx) so far.
As with our investigation of how well we can plan for future decades, we decided to publish our initial findings after investigating only a few historical cases. This allows us to gain feedback on the value of the project, as well as suggestions for improvement, before continuing. It also means that we aren't yet able to draw any confident conclusions about our core questions.
The most significant results from this project so far are:
- We came up with a preliminary list of 6 seemingly-important ways in which a historical case could be analogous to the future invention of AGI, and evaluated several historical cases on these criteria.
- Climate change risk seems sufficiently disanalogous to AI risk that studying climate change mitigation efforts probably gives limited insight into how well policy-makers will deal with AGI risk: the expected damage of climate change appears to be very small relative to the the expected damage due to AI risk, especially when one looks at expected damage to policy makers.
- The 2008 financial crisis appears, after a shallow investigation, to be sufficiently analogous to AGI risk that it should give us some small reason to be concerned that policy-makers will not manage the invention of AGI wisely.
- The risks to critical infrastructure from geomagnetic storms are far too small to be in the same reference class with risks from AGI.
- The eradication of smallpox is only somewhat analogous to the invention of AGI.
- Jonah performed very shallow investigations of how policy-makers have handled risks from cyberwarfare, chlorofluorocarbons, and the Cuban missile crisis, but these cases need more study before even "initial thoughts" can be given.
- We identified additional historical cases that could be investigated in the future.
Further details are given below. For sources and more, please see our full email exchange (.docx).
6 ways a historical case can be analogous to the invention of AGI
In conversation, Jonah and I identified six features of the future invention of AGI that, if largely shared by a historical case, seem likely to allow the historical case to shed light on how well policy-makers will deal with the invention of AGI:
- AGI may become a major threat in a somewhat unpredictable time.
- AGI may become a threat when the world has very limited experience with it.
- A good outcome with AGI may require solving a difficult global coordination problem.
- Preparing for the AGI threat adequately may require lots of careful work in advance.
- Policy-makers have strong personal incentives to solve the AGI problem.
- A bad outcome with AGI would be a global disaster, and a good outcome with AGI would have global humanitarian benefit.
More details on these criteria and their use are given in the second email of our full email exchange.
Risks from climate change
People began to see climate change as a potential problem in the early 1970s, but there was some ambiguity as to whether human activity was causing warming (because of carbon emissions) or cooling (because of smog particles). The first IPCC report was issued in 1990, and stated that were was substantial anthropogenic global warming due to greenhouse gases. By 2001, there was a strong scientific consensus behind this claim. While policy-makers' response to risks from climate change might seem likely to shed light on whether policy-makers will deal wisely with AGI, there are some important disanalogies:
- The harms of global warming are expected to fall disproportionately on disadvantaged people in poor countries, not on policy-makers. So policy-makers have much less personal incentive to solve the problem than is the case with AGI.
- In the median case, humanitarian losses from global warming seems to be about 20% of GDP per year for the poorest people. In light of anticipated economic development and marginal diminishing utility, this is a much smaller negative humanitarian impact than AGI risk (even ignoring future generations). For example, economist Indur Goklany estimated that "through 2085, only 13% of [deaths] from hunger, malaria, and extreme weather events (including coastal flooding from sea level rise) should be from [global] warming."
- Thus, potential analogies to AGI risk come from climate change's tail risk. But there seem to be few credentialed scientists who have views compatible with a prediction that even a temperature increase in the 95th percentile of the probability distribution (by 2100) would do more than just begin to render some regions of Earth uninhabitable.
- According to the 5th IPCC, the risk of human extinction from climate change seems very low: "Some thresholds that all would consider dangerous have no support in the literature as having a non-negligible chance of occurring. For instance, a 'runaway greenhouse effect'—analogous to Venus—appears to have virtually no chance of being induced by anthropogenic activities."
The 2008 financial crisis
Jonah did a shallow investigation of the 2008 financial crisis, but the preliminary findings are interesting enough for us to describe them in some detail. Jonah's impressions about the relevance of the 2008 financial crisis to the AGI situation are based on a reading of After the Music Stopped by Alan Blinder, who was the vice chairman of the federal reserve for 1.5 years during the Clinton administration. Naturally, many additional sources should be consulted before drawing firm conclusions about the relevance of policy-makers' handling of the financial crisis to their likelihood of handling AGI wisely.
Blinder's seven main factors leading to the recession are (p. 27):
- Inflated asset prices, especially of houses (the housing bubble) but also of certain securities (the bond bubble);
- Excessive leverage (heavy borrowing) throughout the financial system and the economy;
- Lax financial regulation, both in terms of what the law left unregulated and how poorly the various regulators performed their duties;
- Disgraceful banking practices in subprime and other mortgage lending;
- The crazy-quilt of unregulated securities and derivatives that were built on these bad mortgages;
- The abysmal performance of the statistical rating agencies, which helped the crazy-quilt get stitched together; and
- The perverse compensation systems in many financial institutions that created powerful incentives to go for broke.
With these factors in mind, let's look at the strength of the analogy between the 2008 financial crisis and the future invention of AGI:
- Almost tautologically, a financial crisis is unexpected, though we do know that financial crises happen with some regularity.
- The 2008 financial crisis was not unprecedented in kind, only in degree (in some ways).
- Avoiding the 2008 financial crisis would have required solving a difficult national coordination problem, rather than a global coordination problem. Still, this analogy seems fairly strong. As Jonah writes, "While the 2008 financial crisis seems to have been largely US specific (while having broader ramifications), there's a sense in which preventing it would have required solving a difficult coordination problem. The causes of the crisis are diffuse, and responsibility falls on many distinct classes of actors."
- Jonah's analysis wasn't deep enough to discern whether the 2008 financial crisis is analogous to the future invention of AGI with regard to how much careful work would have been required in advance to avert the risk.
- In contrast with AI risk, the financial crisis wasn't a life or death matter for almost any of the actors involved. Many people in finance didn't have incentives to avert the financial crisis: indeed, some of the key figures involved were rewarded with large bonuses. But it's plausible that government decision makers had incentive to avert a financial crisis for reputational reasons, and many interest groups are adversely affected by financial crises.
- Once again, the scale of the financial crisis wasn't on a par with AI risk, but it was closer to that scale than the other risks Jonah looked at in this initial investigation.
Jonah concluded that "the conglomerate of poor decisions [leading up to] the 2008 financial crisis constitute a small but significant challenge to the view that [policy-makers] will successfully address AI risk." His reasons were:
- The magnitude of the financial crisis is nontrivial (even if small) compared with the magnitude of the AI risk problem (not counting future generations).
- The financial crisis adversely affected a very broad range of people, apparently including a large fraction of those people in positions of power (this seems truer here than in the case of climate change). A recession is bad for most businesses and for most workers. Yet these actors weren't able to recognize the problem, coordinate, and prevent it.
- The reasons that policy-makers weren't able to recognize the problem, coordinate, and prevent it seem related to reasons why people might not recognize AI risk as a problem, coordinate, and prevent it. First, several key actors involved seem to have exhibited conspicuous overconfidence and neglect of tail risk (e.g. Summers, etc. ignoring Brooksley Born's warnings about excessive leverage). If true, this shows that people in positions of power are notably susceptible to overconfidence and neglect of tail risk. Avoiding overconfidence and giving sufficient weight to tail risk may be crucial in mitigating AI risk. Second, one gets a sense that bystander effect and tragedy of the commons played a large role in the case of the financial crisis. There are risks that weren't adequately addressed because doing so didn't fall under the purview of any of the existing government agencies. This may have corresponded to a mentality of the type "that's not my job — somebody else can take care of it." If people think that AI risk is large, then they might think "if nobody's going to take care of it then I will, because otherwise I'm going to die." But if people think that AI risk is small, they might think "This probably won't be really bad for me, and even though someone should take care of it, it's not going to be me."
Risks from geomagnetic storms
Large geomagnetic storms like the 1859 Carrington Event are infrequent, but could cause serious damage to satellites and critical infrastructure. See this OECD report for an overview.
Jonah's investigation revealed a wide range in expected losses from geomagnetic storms, from $30 million per year to $30 billion per year. But even this larger number amounts to $1.5 trillion in expected losses over the next 50 years. Compare this with the losses from the 2008 financial crisis (roughly a 1 in 50 years event), which are estimated to be about $13 trillion for Americans alone.
Though serious, the risks from geomagnetic storms appear to be small enough to be disanalogous to the future invention of AGI.
The eradication of smallpox
Smallpox, after killing more than 500 million people over the past several millennia, was eradicated in 1979 after a decades-long global eradication effort. Though a hallmark of successful global coordination, it doesn't seem especially relevant to whether policy-makers will handle the invention of AGI wisely.
Here's how the eradication of smallpox does our doesn't fit our criteria for being analogous to the future invention of AGI:
- Smallpox didn't arrive at an unpredictable time; it arrived millennia before the eradication campaign.
- The world didn't have experience eradicating a disease before smallpox was eradicated, but a number of nations had eliminated smallpox.
- Smallpox eradication required solving a difficult global coordination problem, but in a way disanalogous to the invention of AGI safety (see the other points on this list).
- Preparing for smallpox eradication required effort in advance in some sense, but the effort had mostly already been exerted before the campaign was announced.
- Nations without smallpox had incentive to eradicate smallpox so that they didn't have to spend money to immunize citizens so that the virus would not be (re)-introduced to their countries. For example, in 1968, the United States spent about $100 million on routine smallpox vaccinations.
- Smallpox can be thought of as a global disaster: by 1966, about 2 million people died of smallpox each year.
Shallow investigations of risks from cyberwarfare, chlorofluorocarbons, and the Cuban missile crisis
Jonah's shallow investigation of risks from cyberwarfare revealed that experts disagree significantly about the nature and scope of these risks. It's likely that dozens of hours of research would be required to develop a well-informed model of these risks.
To investigate how policy-makers handled the discovery that chlorofluorocarbons (CFCs) depleted the ozone layer, Jonah summarized the first 100 pages of Ozone Crisis: The 15-Year Evolution of a Sudden Global Emergency (see our full email exchange for the summary). This historical case seems worth investigating further, and may be a case of policy-makers solving a global risk with surprising swiftness, though whether the response was appropriately prompt is debated.
Jonah also did a shallow investigation of the Cuban missile crisis. It's difficult to assess how likely it was for the crisis to escalate into a global nuclear war, but it appears that policy-makers made many poor decisions leading up to and during the Cuban missile crisis (see our full email exchange for a list). Jonah concludes:
even if the probability of the Cuban missile crisis leading to an all out nuclear war was only 1% or so, the risk was still sufficiently great so that the way in which the actors handled the situation is evidence against elites handling the creation of AI well. (This contrasts with the situation with climate change, in that elites had strong personal incentives to avert an all-out nuclear war.)
However, this is only a guess based on a shallow investigation, and should not be taken too seriously before a more thorough investigation of the historical facts can be made.
Additional historical cases that could be investigated
We also identified additional historical cases that could be investigated for potentially informative analogies to the future invention of AGI:
- The 2003 Iraq War
- The frequency with which dictators are deposed or assassinated due to "unforced errors" they made
- Nuclear proliferation
- Recombinant DNA
- Molecular nanotechnology
- Near Earth objects
- Pandemics and potential pandemics (e.g. HIV, SARS)
Just letting alcoholics drink
"Wet houses"-- subsidized housing for alcoholics (they need to get most of their own money for alcohol, but their other expenses are covered) might actually be a good idea. It's cheaper than trying to get them to stop drinking, arguably kinder than trying to get people to take on a very hard task that they aren't interested in, and leads to less collateral damage than having alcoholics couch-surfing or living on the street.
Utilitarians, what do you think?
Scott Sumner on Utility vs Happiness [Link]
A distinction that some people grok right away and some others may not realize exists:
Imagine a country called “Lanmindia,” where much of the population has seen its legs blown off in horrible accidents. Does that sound like a pretty miserable place? Happiness research suggests not. The claim is that there is a sort of natural “set-point” for happiness, and that after winning a lottery one is happy for a short time, and then you revert right back to your natural happiness level. I find that plausible. They also claim that if someone loses a limb, then they are unhappy for a short period and then revert back to normal. I find that implausible, but if the evidence says it is the case then I guess I need to accept that.
My claim is that although Lanmindia is just as happy as America, it has much lower utility. Let’s define ’utility’ as ”that which people maximize.” People very much don’t want to have their legs blown off, and hence emigrate from Lanmindia in droves. People behave as if they care about utility, not happiness.
-Scott Sumner, "Nonsense on stilts: Part 1. What if utility and happiness are unrelated?" TheMoneyIllusion
This is also somewhat a reply to Hanson's "Lift Up Your Eyes" on Overcoming Bias. Some people on LessWrong are careful to make the distinction between ordinal utility, cardinal utility, and fuzzies, and others aren't quite so much. The above sentence on accepting evidence and the post script that he is not serious about one part of the post might also make interesting conversation -- part two is advice to move next door to a child molester for cheaper housing if you don't have a kid and part three is about The Fed taking advantage of banks.
= 783df68a0f980790206b9ea87794c5b6)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)