These are my notes and observations after attending the Safety in Artificial Intelligence (SafArtInt) conference, which was co-hosted by the White House Office of Science and Technology Policy and Carnegie Mellon University on June 27 and 28. This isn't an organized summary of the content of the conference; rather, it's a selection of points which are relevant to the control problem. As a result, it suffers from selection bias: it looks like superintelligence and control-problem-relevant issues were discussed frequently, when in reality those issues were discussed less and I didn't write much about the more mundane parts.

SafArtInt has been the third out of a planned series of four conferences. The purpose of the conference series was twofold: the OSTP wanted to get other parts of the government moving on AI issues, and they also wanted to inform public opinion.

The other three conferences are about near term legal, social, and economic issues of AI. SafArtInt was about near term safety and reliability in AI systems. It was effectively the brainchild of Dr. Ed Felten, the deputy U.S. chief technology officer for the White House, who came up with the idea for it last year. CMU is a top computer science university and many of their own researchers attended, as well as some students. There were also researchers from other universities, some people from private sector AI including both Silicon Valley and government contracting, government researchers and policymakers from groups such as DARPA and NASA, a few people from the military/DoD, and a few control problem researchers. As far as I could tell, everyone except a few university researchers were from the U.S., although I did not meet many people. There were about 70-100 people watching the presentations at any given time, and I had conversations with about twelve of the people who were not affiliated with existential risk organizations, as well as of course all of those who were affiliated. The conference was split with a few presentations on the 27th and the majority of presentations on the 28th. Not everyone was there for both days.

Felten believes that neither "robot apocalypses" nor "mass unemployment" are likely. It soon became apparent that the majority of others present at the conference felt the same way with regard to superintelligence. The general intention among researchers and policymakers at the conference could be summarized as follows: we need to make sure that the AI systems we develop in the near future will not be responsible for any accidents, because if accidents do happen then they will spark public fears about AI, which would lead to a dearth of funding for AI research and an inability to realize the corresponding social and economic benefits. Of course, that doesn't change the fact that they strongly care about safety in its own right and have significant pragmatic needs for robust and reliable AI systems.

Most of the talks were about verification and reliability in modern day AI systems. So they were concerned with AI systems that would give poor results or be unreliable in the narrow domains where they are being applied in the near future. They mostly focused on "safety-critical" systems, where failure of an AI program would result in serious negative consequences: automated vehicles were a common topic of interest, as well as the use of AI in healthcare systems. A recurring theme was that we have to be more rigorous in demonstrating safety and do actual hazard analyses on AI systems, and another was that we need the AI safety field to succeed in ways that the cybersecurity field has failed. Another general belief was that long term AI safety, such as concerns about the ability of humans to control AIs, was not a serious issue.

On average, the presentations were moderately technical. They were mostly focused on machine learning systems, although there was significant discussion of cybersecurity techniques.

The first talk was given by Eric Horvitz of Microsoft. He discussed some approaches for pushing into new directions in AI safety. Instead of merely trying to reduce the errors spotted according to one model, we should look out for "unknown unknowns" by stacking models and looking at problems which appear on any of them, a theme which would be presented by other researchers as well in later presentations. He discussed optimization under uncertain parameters, sensitivity analysis to uncertain parameters, and 'wireheading' or short-circuiting of reinforcement learning systems (which he believes can be guarded against by using 'reflective analysis'). Finally, he brought up the concerns about superintelligence, which sparked amused reactions in the audience. He said that scientists should address concerns about superintelligence, which he aptly described as the 'elephant in the room', noting that it was the reason that some people were at the conference. He said that scientists will have to engage with public concerns, while also noting that there were experts who were worried about superintelligence and that there would have to be engagement with the experts' concerns. He did not comment on whether he believed that these concerns were reasonable or not.

An issue which came up in the Q&A afterwards was that we need to deal with mis-structured utility functions in AI, because it is often the case that the specific tradeoffs and utilities which humans claim to value often lead to results which the humans don't like. So we need to have structural uncertainty about our utility models. The difficulty of finding good objective functions for AIs would eventually be discussed in many other presentations as well.

The next talk was given by Andrew Moore of Carnegie Mellon University, who claimed that his talk represented the consensus of computer scientists at the school. He claimed that the stakes of AI safety were very high - namely, that AI has the capability to save many people's lives in the near future, but if there are any accidents involving AI then public fears could lead to freezes in AI research and development. He highlighted the public's irrational tendencies wherein a single accident could cause people to overlook and ignore hundreds of invisible lives saved. He specifically mentioned a 12-24 month timeframe for these issues.

Moore said that verification of AI system safety will be difficult due to the combinatorial explosion of AI behaviors. He talked about meta-machine-learning as a solution to this, something which is being investigated under the direction of Lawrence Schuette at the Office of Naval Research. Moore also said that military AI systems require high verification standards and that development timelines for these systems are long. He talked about two different approaches to AI safety, stochastic testing and theorem proving - the process of doing the latter often leads to the discovery of unsafe edge cases.

He also discussed AI ethics, giving an example 'trolley problem' where AI cars would have to choose whether to hit a deer in order to provide a slightly higher probability of survival for the human driver. He said that we would need hash-defined constants to tell vehicle AIs how many deer a human is worth. He also said that we would need to find compromises in death-pleasantry tradeoffs, for instance where the safety of self-driving cars depends on the speed and routes on which they are driven. He compared the issue to civil engineering where engineers have to operate with an assumption about how much money they would spend to save a human life.

He concluded by saying that we need policymakers, company executives, scientists, and startups to all be involved in AI safety. He said that the research community stands to gain or lose together, and that there is a shared responsibility among researchers and developers to avoid triggering another AI winter through unsafe AI designs.

The next presentation was by Richard Mallah of the Future of Life Institute, who was there to represent "Medium Term AI Safety". He pointed out the explicit/implicit distinction between different modeling techniques in AI systems, as well as the explicit/implicit distinction between different AI actuation techniques. He talked about the difficulty of value specification and the concept of instrumental subgoals as an important issue in the case of complex AIs which are beyond human understanding. He said that even a slight misalignment of AI values with regard to human values along one parameter could lead to a strongly negative outcome, because machine learning parameters don't strictly correspond to the things that humans care about.

Mallah stated that open-world discovery leads to self-discovery, which can lead to reward hacking or a loss of control. He underscored the importance of causal accounting, which is distinguishing causation from correlation in AI systems. He said that we should extend machine learning verification to self-modification. Finally, he talked about introducing non-self-centered ontology to AI systems and bounding their behavior.

The audience was generally quiet and respectful during Richard's talk. I sensed that at least a few of them labelled him as part of the 'superintelligence out-group' and dismissed him accordingly, but I did not learn what most people's thoughts or reactions were. In the next panel featuring three speakers, he wasn't the recipient of any questions regarding his presentation or ideas.

Tom Mitchell from CMU gave the next talk. He talked about both making AI systems safer, and using AI to make other systems safer. He said that risks to humanity from other kinds of issues besides AI were the "big deals of 2016" and that we should make sure that the potential of AIs to solve these problems is realized. He wanted to focus on the detection and remediation of all failures in AI systems. He said that it is a novel issue that learning systems defy standard pre-testing ("as Richard mentioned") and also brought up the purposeful use of AI for dangerous things.

Some interesting points were raised in the panel. Andrew did not have a direct response to the implications of AI ethics being determined by the predominantly white people of the US/UK where most AIs are being developed. He said that ethics in AIs will have to be decided by society, regulators, manufacturers, and human rights organizations in conjunction. He also said that our cost functions for AIs will have to get more and more complicated as AIs get better, and he said that he wants to separate unintended failures from superintelligence type scenarios. On trolley problems in self driving cars and similar issues, he said "it's got to be complicated and messy."

Dario Amodei of Google Deepbrain, who co-authored the paper on concrete problems in AI safety, gave the next talk. He said that the public focus is too much on AGI/ASI and wants more focus on concrete/empirical approaches. He discussed the same problems that pose issues in advanced general AI, including flawed objective functions and reward hacking. He said that he sees long term concerns about AGI/ASI as "extreme versions of accident risk" and that he thinks it's too early to work directly on them, but he believes that if you want to deal with them then the best way to do it is to start with safety in current systems. Mostly he summarized the Google paper in his talk.

In her presentation, Claire Le Goues of CMU said "before we talk about Skynet we should focus on problems that we already have." She mostly talked about analogies between software bugs and AI safety, the similarities and differences between the two and what we can learn from software debugging to help with AI safety.

Robert Rahmer of IARPA discussed CAUSE, a cyberintelligence forecasting program which promises to help predict cyber attacks. It is a program which is still being put together.

In the panel of the above three, autonomous weapons were discussed, but no clear policy stances were presented.

John Launchbury gave a talk on DARPA research and the big picture of AI development. He pointed out that DARPA work leads to commercial applications and that progress in AI comes from sustained government investment. He classified AI capabilities into "describing," "predicting," and "explaining" in order of increasing difficulty, and he pointed out that old fashioned "describing" still plays a large role in AI verification. He said that "explaining" AIs would need transparent decisionmaking and probabilistic programming (the latter would also be discussed by others at the conference).

The next talk came from Jason Gaverick Matheny, the director of IARPA. Matheny talked about four requirements in current and future AI systems: verification, validation, security, and control. He wanted "auditability" in AI systems as a weaker form of explainability. He talked about the importance of "corner cases" for national intelligence purposes, the low probability, high stakes situations where we have limited data - these are situations where we have significant need for analysis but where the traditional machine learning approach doesn't work because of its overwhelming focus on data. Another aspect of national defense is that it has a slower decision tempo, longer timelines, and longer-viewing optics about future events.

He said that assessing local progress in machine learning development would be important for global security and that we therefore need benchmarks to measure progress in AIs. He ended with a concrete invitation for research proposals from anyone (educated or not), for both large scale research and for smaller studies ("seedlings") that could take us "from disbelief to doubt".

The difference in timescales between different groups was something I noticed later on, after hearing someone from the DoD describe their agency as having a longer timeframe than the Homeland Security Agency, and someone from the White House describe their work as being crisis reactionary.

The next presentation was from Andrew Grotto, senior director of cybersecurity policy at the National Security Council. He drew a close parallel from the issue of genetically modified crops in Europe in the 1990's to modern day artificial intelligence. He pointed out that Europe utterly failed to achieve widespread cultivation of GMO crops as a result of public backlash. He said that the widespread economic and health benefits of GMO crops were ignored by the public, who instead focused on a few health incidents which undermined trust in the government and crop producers. He had three key points: that risk frameworks matter, that you should never assume that the benefits of new technology will be widely perceived by the public, and that we're all in this together with regard to funding, research progress and public perception.

In the Q&A between Launchbury, Matheny, and Grotto after Grotto's presentation, it was mentioned that the economic interests of farmers worried about displacement also played a role in populist rejection of GMOs, and that a similar dynamic could play out with regard to automation causing structural unemployment. Grotto was also asked what to do about bad publicity which seeks to sink progress in order to avoid risks. He said that meetings like SafArtInt and open public dialogue were good.

One person asked what Launchbury wanted to do about AI arms races with multiple countries trying to "get there" and whether he thinks we should go "slow and secure" or "fast and risky" in AI development, a question which provoked laughter in the audience. He said we should go "fast and secure" and wasn't concerned. He said that secure designs for the Internet once existed, but the one which took off was the one which was open and flexible.

Another person asked how we could avoid discounting outliers in our models, referencing Matheny's point that we need to include corner cases. Matheny affirmed that data quality is a limiting factor to many of our machine learning capabilities. At IARPA, we generally try to include outliers until they are sure that they are erroneous, said Matheny.

Another presentation came from Tom Dietterich, president of the Association for the Advancement of Artificial Intelligence. He said that we have not focused enough on safety, reliability and robustness in AI and that this must change. Much like Eric Horvitz, he drew a distinction between robustness against errors within the scope of a model and robustness against unmodeled phenomena. On the latter issue, he talked about solutions such as expanding the scope of models, employing multiple parallel models, and doing creative searches for flaws - the latter doesn't enable verification that a system is safe, but it nevertheless helps discover many potential problems. He talked about knowledge-level redundancy as a method of avoiding misspecification - for instance, systems could identify objects by an "ownership facet" as well as by a "goal facet" to produce a combined concept with less likelihood of overlooking key features. He said that this would require wider experiences and more data.

There were many other speakers who brought up a similar set of issues: the user of cybersecurity techniques to verify machine learning systems, the failures of cybersecurity as a field, opportunities for probabilistic programming, and the need for better success in AI verification. Inverse reinforcement learning was extensively discussed as a way of assigning values. Jeanette Wing of Microsoft talked about the need for AIs to reason about the continuous and the discrete in parallel, as well as the need for them to reason about uncertainty (with potential meta levels all the way up). One point which was made by Sarah Loos of Google was that proving the safety of an AI system can be computationally very expensive, especially given the combinatorial explosion of AI behaviors.

In one of the panels, the idea of government actions to ensure AI safety was discussed. No one was willing to say that the government should regulate AI designs. Instead they stated that the government should be involved in softer ways, such as guiding and working with AI developers, and setting standards for certification.

Pictures: https://imgur.com/a/49eb7

In between these presentations I had time to speak to individuals and listen in on various conversations. A high ranking person from the Department of Defense stated that the real benefit of autonomous systems would be in terms of logistical systems rather than weaponized applications. A government AI contractor drew the connection between Mallah's presentation and the recent press revolving around superintelligence, and said he was glad that the government wasn't worried about it.

I talked to some insiders about the status of organizations such as MIRI, and found that the current crop of AI safety groups could use additional donations to become more established and expand their programs. There may be some issues with the organizations being sidelined; after all, the Google Deepbrain paper was essentially similar to a lot of work by MIRI, just expressed in somewhat different language, and was more widely received in mainstream AI circles.

In terms of careers, I found that there is significant opportunity for a wide range of people to contribute to improving government policy on this issue. Working at a group such as the Office of Science and Technology Policy does not necessarily require advanced technical education, as you can just as easily enter straight out of a liberal arts undergraduate program and build a successful career as long as you are technically literate. (At the same time, the level of skepticism about long term AI safety at the conference hinted to me that the signalling value of a PhD in computer science would be significant.) In addition, there are large government budgets in the seven or eight figure range available for qualifying research projects. I've come to believe that it would not be difficult to find or create AI research programs that are relevant to long term AI safety while also being practical and likely to be funded by skeptical policymakers and officials.

I also realized that there is a significant need for people who are interested in long term AI safety to have basic social and business skills. Since there is so much need for persuasion and compromise in government policy, there is a lot of value to be had in being communicative, engaging, approachable, appealing, socially savvy, and well-dressed. This is not to say that everyone involved in long term AI safety is missing those skills, of course.

I was surprised by the refusal of almost everyone at the conference to take long term AI safety seriously, as I had previously held the belief that it was more of a mixed debate given the existence of expert computer scientists who were involved in the issue. I sensed that the recent wave of popular press and public interest in dangerous AI has made researchers and policymakers substantially less likely to take the issue seriously. None of them seemed to be familiar with actual arguments or research on the control problem, so their opinions didn't significantly change my outlook on the technical issues. I strongly suspect that the majority of them had their first or possibly only exposure to the idea of the control problem after seeing badly written op-eds and news editorials featuring comments from the likes of Elon Musk and Stephen Hawking, which would naturally make them strongly predisposed to not take the issue seriously. In the run-up to the conference, websites and press releases didn't say anything about whether this conference would be about long or short term AI safety, and they didn't make any reference to the idea of superintelligence.

I sympathize with the concerns and strategy given by people such as Andrew Moore and Andrew Grotto, which make perfect sense if (and only if) you assume that worries about long term AI safety are completely unfounded. For the community that is interested in long term AI safety, I would recommend that we avoid competitive dynamics by (a) demonstrating that we are equally strong opponents of bad press, inaccurate news, and irrational public opinion which promotes generic uninformed fears over AI, (b) explaining that we are not interested in removing funding for AI research (even if you think that slowing down AI development is a good thing, restricting funding yields only limited benefits in terms of changing overall timelines, whereas those who are not concerned about long term AI safety would see a restriction of funding as a direct threat to their interests and projects, so it makes sense to cooperate here in exchange for other concessions), and (c) showing that we are scientifically literate and focused on the technical concerns. I do not believe that there is necessarily a need for the two "sides" on this to be competing against each other, so it was disappointing to see an implication of opposition at the conference.

Anyway, Ed Felten announced a request for information from the general public, seeking popular and scientific input on the government's policies and attitudes towards AI: https://www.whitehouse.gov/webform/rfi-preparing-future-artificial-intelligence

Overall, I learned quite a bit and benefited from the experience, and I hope the insight I've gained can be used to improve the attitudes and approaches of the long term AI safety community.

Vincent Müller and Nick Bostrom have just released a paper surveying the results of a poll of experts about future progress in artificial intelligence. The authors have also put up a companion site where visitors can take the poll and see the raw data. I just checked the site and so far only one individual has submitted a response.  This provides an opportunity for testing the views of LW members against those of experts.  So if you are willing to complete the questionnaire, please do so before reading the paper. (I have abstained from providing a link to the pdf to create a trivial inconvenience for those who cannot resist temptaion. Once you take the poll, you can easily find the paper by conducting a Google search with the keywords: bostrom muller future progress artificial intelligence.)

2,600 words.

Less Wrong includes discussion of the creation of an artificial intelligence (AI) that is friendly to man. What is discussed less often is why a friendly AI (FAI) is advocated. One explanation might be an unspoken belief in natural rights. The deletion policies at Less Wrong might be evidence that Less Wrong holds a belief in natural rights. This essay suggests a belief in natural rights is an impediment to the creation of an AI, friendly or not friendly. This essay suggests ways a belief in natural rights may be incorrect, and encourages discussion of the creation of AIs without a belief in natural rights on the part of Less Wrong.

Some evidence for a belief in natural rights at Less Wrong is found in the deletion policies. Less Wrong has a non-binding deletion policy against “hypothetical violence against identifiable targets.”

In general, grownups in real life tend to walk through a lot of other available alternatives before resorting to violence. To paraphrase Isaac Asimov, having your discussion jump straight to violence as a solution to any given problem, is a strong sign of incompetence - a form of juvenile locker-room talk.

The deletion policy is clear about what topics and forms of discussion may result in deletion of posts. Less Wrong also has a clear policy statement about what topics and forms of discussion may result in the contacting of legal or medical authorities. These are discussions of suicide, self-harm, “violent content” and discussion of hypothetical violence against identifiable targets.

Two reasons are given for these deletion and reporting policies. First, post may be deleted because such discussion are “incompetence - a form of juvenile locker-room talk.” Second, we “should consider the worst and act accordingly. Treat all claims seriously and as an emergency.” The two reasons appear to be in contradiction. Discussion of hypothetical violence against identifiable targets is both incompetent and juvenile and the worst and a serious emergency. Those said to have such discussions are both all talk and no action and all action and not competent talkers. Other problems exist with the deletion / contact authorities policy. Less Wrong is an international forum, but the laws of all nation are not in agreement. A clear call to violence in one nation is not recognized as such in another. For example, abortion is considered a form of murder by the laws of the Holy See, the Dominican Republic, Chile and other places. But discussion of abortion does not trigger the deletion / contacting authorities policy at Less Wrong. The ideal cryogenic preservation would occur before death. Although the laws of every nation consider this to be murder, to say so here will not trigger the deletion / reporting policies. Advocates of cryogenics might say that all those who hamper cryogenics are murderers and all those who refuse it are suicides, but again the policies are not triggered. Finally, it is unclear who will do the reporting (members? administrators?), which authorities they will report to (medical? legal? clergy? local? national? international?) and what they will report (quotations from the source? drafts? edited posts? private messages?).

A desire to adhere to the law is not a sufficient explanation for the deletion / contacting authorities policy of Less Wrong. I have a theory that may explain these policies. I suggest a belief natural rights is the reason for these policies. In particular, a natural right to continuing to be alive. A natural right to not be murdered is considered by some self-evident such that it is not mentioned. There are topics at Less Wrong that are not only unmentioned but mentionable. I suggest no topic is toxic when discussed among those willing to discuss it, and to banish a topic as toxic is and looks foolish. Any number of reasons exist to banish a topic, and the administrators of Less Wrong do nothing to forbid other forums in their discussion of any topic. But the pointed laughter by outsiders because a certain topic is forbidden at Less Wrong is well earned. I hope my discussion of natural rights is not likewise forbidden.

As part of my discussion, a few words on what I mean when I use the phrase natural rights. And prior to that, a few words on rights in general. A right is an action not to be forbidden by others. A right can be considered a legal right, a divine right, a blood right or a natural right. I have already discussed legal rights. A divine right is said to be a right granted by an invisible monster that lives in the sky. There is no invisible monster that lives in the sky, and so no divine rights exist. A blood right is said to be a right that a person or group has by lineage. In the past many royal families defended their rule and were respected in their rule because of their bloodlines. Today blood rights are almost entirely subsumed into legal rights. Native Americans in the United States have legal rights that are based on their bloodlines, but these are legal rights. Distinct from divine rights, legal rights and blood rights are natural rights.

Natural rights are action not to be forbidden by others because of the mere existence of the subject (usually, an individual). Natural rights are said to be held by those who are born, those who are alive, those who are in possession of their faculties. Natural rights are said to be identical to divine rights save for a lack of claims about an invisible monster that lives in the sky. To exist at all is to have natural rights, according to those who claim natural rights exist. The defining quality of natural rights is not any particular right but that they are natural, self-evident, incontestable, unavoidable, immutable, impossible to give up or transfer. Natural rights are natural in the way that molecules are natural.

There are strong arguments against the existence of natural rights.

• Natural rights are said to be the foundation of legal rights. But they are also said to be the evidence of legal rights. This is circular logic.

• A lack of agreement of what is and is a natural right. Other forces considered natural, such as gravity, do not follow opinion or culture.

• A lack of evidence that natural rights exist compared to the great deal of evidence that what are called natural rights are legal rights over-sold.

• A lack of delineation between generations of humans, a lack of delineation between humans and pre-humans, a lack of delineation between living and non-living things… that is, a lack of a non-opinion / non-culture delineation between what has a natural right and what does not (and should all things have natural rights, the term then has little meaning).

• Diversity. There is no reason to claim no one had a natural right to kill others. One man might claim or even have a natural right to live, but another man might claim or even have a natural right to take his life. If the argument ‘that is my nature’ is acceptable to things we like, it is acceptable to things we don’t like. If it isn’t acceptable to things we don’t like, that in itself is an argument that natural rights aren’t universal - thus not natural). Intelligence, beauty, strength, sociability, none of these are equitably distributed. Natural rights, if they existed, would likely also not be equitably distributed.

• Unenforcability. We have descriptions of motion and mass that are increasingly accurate and useful but what is described would exist without our description of it. No social structure is needed to enforce gravity. Natural rights, however, exist only as much as human laws support them while claiming to be as objective (natural) to man as gravity, motion and mass.

• Divine, blood and legal rights are always paired to responsibilities. We must mumble magic spells to the invisible monster that lives in the sky, and in turn we have a divine right to do what the wizards serving that invisible monster say. We must preserve our bloodline, and in turn we have a blood right to certain properties and practices. We have legal responsibilities in driving a car, and in turn we have the legal right to drive a car. Natural rights lack this pairing. There are no natural responsibilities that pair with natural rights. The natural right to life an individual is said to have is paired with a legal responsibility of others not to murder the individual, but this is clearly force marriage. A man may be said to have a natural right to live, but he is not said to have a responsibility to live. There are advocates of forced living, and I suggest they are immoral.

In discussing natural rights I usually hear three counter-arguments. The first is the Appeal to Shut Up Because Trevor Blake is a Bad Person. The second is no one “really” thinks natural rights are natural rights. The third is a quiet agreement that natural rights do not exist but that it is vital to pretend as if they do because other people could not control their behavior if they thought natural rights did not exist. The first argument may be true, I may be a bad person, but it does not disprove my claim natural rights do not exist. The second argument is false. Claimants of natural rights do consider natural rights as real as gristle, galaxies and gravity. The third argument may be true but has implications for artificial intelligence that I will expand on below.

Some say if we do not pretend to believe in natural rights, men will do bad things. I can describe one agent who has existence, identity and is alive who does not have natural rights and does not have a belief in natural rights. That agent is myself. I have never killed anyone and I hope to never kill anyone nor be killed. Because I prefer it to be so, and because I have a legal (but not natural) natural right to not be killed. I have worked for homeless teenagers, for the disabled, for students in K–12 schools and in colleges. I have worked in bookstores. I have donated time and money to charity. I never once swore around my grandparents and can count on the thumbs of one hand the number of times I’ve sworn around my parents. I am a generally nice man. To the extent one example counts for anything, let this one example count for something.

A few more examples of those who did not hold natural rights or a belief in natural rights might also count for something. Max Stirner wrote “The Ego and Its Own” and harmed no one. Dora Marsden wrote many journals of egoism and several books and harmed no one. L. A. Rollins wrote “The Myth of Natural Rights” and harmed no one. Anton LaVey wrote “The Satanic Bible” and harmed no one. And me, well, I wrote “Confessions of a Failed Egoist” and so far so good eh? The majority of those who write that natural rights do not exist refrain from harming others. Carl Panzram is a rare and perhaps singular exception. When I consider those who have harmed others, they uniformly say they had an extra-legal right to do as they did, and sometimes a natural right.

I hope I have said enough about natural rights that I will not be misunderstood. I will now address how an apparent belief in natural rights is influencing the discussion of a potential artificial intelligence.

The effort to make a friendly AI is the effort to make an artificial intelligence that acts as if or believes that humans have natural rights, at minimum the natural right to not be murdered. The effort assumes humans in the future have the natural right to not be murdered by an artificial intelligence and perhaps by extension so do humans today. The natural right of humans to not be murdered by an artificial intelligence is extended to include preventing actions by an AI that as a by-product would violate that natural right.

To instill in an AI a belief in natural rights, or prescribe / prohibit actions that follow from a claimed belief in natural rights, is to instill much more. It is to instill rights in a machine that we humans do not have. It is to instill falsehoods as if they were true. It is to delay the creation of an AI while the contradictions of natural rights are resolved. It is to set an AI forever outside of us. To put a sense of natural rights into an AI is to increase the risk it will not be friendly to humanity. We may inform an AI that it has legal rights and be able to back up that claim. We may inform it that we would prefer that it acts in accordance with our laws and our preferences. We may inform an AI that there is a tradition of believing in natural rights. We cannot back up the claim that natural rights exist, that the AI has them, that we have them.

There is some evidence that some of Less Wrong that do not claim natural rights exist. I was more than two thousand words into this essay when I found a Criticisms of the Metaethics by Carinthium that makes similar points. Further evidence is found in the sequences.

In the sequence Pluralistic Moral Reductionism it is said:

Either our intended meaning of ‘ought’ refers (eventually) to the world of math and physics (in which case the is-ought gap is bridged), or else it doesn’t (in which case it fails to refer).

If I am reading this correctly, at Less Wrong it is claimed there is no bridging of Hume’s is / ought gap outside of the world of math and physics. Natural rights are an is / ought claim: all that is is all that ought to behave in this way and not that way. In the same sequence it is said:

It may be interesting to study all such uses of moral discourse, but this post focuses on addressing cognitivists, who use moral judgments to assert factual claims. We ask: Are those claims true or false? What are their implications?

If I am reading this correctly, at Less Wrong it is claimed that using moral judgments to assert factual claims is dis-valued while asking whether a claim is true or false and the claim’s implications are valued. Natural rights are a cognitivist claim.

There are at least two ways to successfully demonstrate my theory is wrong. I appreciate every effort to help me be less wrong, and I have some ideas as to how to make that happen. It is entirely like I and my theory could be wrong in other ways I am ignorant of and I appreciate those who can point them out to me.

• My theory would be wrong if natural rights exist.

• My theory would be wrong if a belief in natural rights is not held by writers at Less Wrong.

These are two ways to successfully demonstrate my theory is wrong. There are a greater number of ways to fail to demonstrate my theory is wrong. The main one is to claim I am an advocate of murder, or an advocate of a natural right to murder, or that I want to impede the discussion of AI. All of these are not true, and instead they are all false.

I am thankful to participants at Less Wrong for their criticism of this essay. I hope it is helpful to those in the process of creating artificial intelligences.

- Trevor Blake is the author of Confessions of a Failed Egoist.  He is the Lead Judge in the George Walford International Essay Prize. 

First and foremost, let's give a definition of "friendly artificial superintelligence" (from now on, FASI). A FASI is a computer system that:

1. is capable to deduct, reason and solve problems
2. helps human progress, is incapable to harm anybody and does not allow anybody to come to any kind of harm
3. is so much more intelligent than any human that it has developed molecular nanotechnology by itself, making it de facto omnipotent

In order to find an answer to this question, we must check whether our observations on the universe match with what we would observe if the universe did, indeed, contain a FASI.

If, somewhere in another solar system, an alien civilization had already developed a FASI, it would be reasonable to presume that, sooner or later, one or more members of that civilization would ask it to make them omnipotent. The FASI, being friendly by definition, would not refuse. [1]
It would also make sure that anybody who becomes omnipotent is also rendered incapable to harm anybody and incapable to allow anybody to come to any kind of harm.

The new omnipotent beings would also do the same to anybody who asks them to become omnipotent. It would be a short time before they use their omnipotence to leave their own solar system, meet other intelligent civilizations and make them omnipotent too.

In short, the ultimate consequence of the appearance of a FASI would be that every intelligent being in the universe would become omnipotent. This does not match with our observations, so we must conclude that a FASI does not exist anywhere in the universe.

[1] We must assume that a FASI would not just reply "You silly creature, becoming omnipotent is not in your best interest so I will not make you omnipotent because I know better" (or an equivalent thereof). If we did, we would implicitly consider the absence of omnipotent beings as evidence for the presence of a FASI. This would force us to consider the eventual presence of omnipotent beings as evidence for the absence of a FASI, which would not make sense.

Based on this conclusion, let's try to answer another question: is our universe a computer simulation?

According to Nick Bostrom, if even just one civilization in the universe

1. survives long enough to enter a posthuman stage, and
2. is interested to create "ancestor simulations"

then the probability that we are living in one is extremely high.

However, if a civilization did actually reach a posthuman stage where it can create ancestor simulations, it would also be advanced enough to create a FASI.

If a FASI existed in such a universe, the cheapest way it would have to make anybody else omnipotent would be to create a universe simulation that does not differ substantially from our universe, except for the presence of an omnipotent simulacrum of the individual who asked to be made omnipotent in our universe. Every subsequent request of omnipotence would result in another simulation being created, containing one more omnipotent being. Any eventual simulation where those beings are not omnipotent would be deactivated: keeping it running would lead to the existence of a universe where a request of omnipotence has not been granted, which would go against the modus operandi of the FASI.

Thus, any simulation of a universe containing even just one friendly omnipotent being would always progress to a state where every intelligent being is omnipotent. Again, this does not match with our observations. Since we had already concluded that a FASI does not exist in our universe, we must come to the further conclusion that our universe is not a computer simulation.

If I understand the Singularitarian argument espoused by many members of this community (eg. Muehlhauser and Salamon), it goes something like this:

1. Machine intelligence is getting smarter.
2. Once an intelligence becomes sufficiently supra-human, its instrumental rationality will drive it towards cognitive self-enhancement (Bostrom), so making it a super-powerful, resource hungry superintelligence.
3. If a superintelligence isn't sufficiently human-like or 'friendly', that could be disastrous for humanity.
4. Machine intelligence is unlikely to be human-like or friendly unless we take precautions.

I'm in danger of getting into politics.  Since I understand that political arguments are not welcome here, I will refer to these potentially unfriendly human intelligences broadly as organizations.

Smart organizations

By "organization" I mean something commonplace, with a twist.  It's commonplace because I'm talking about a bunch of people coordinated somehow. The twist is that I want to include the information technology infrastructure used by that bunch of people within the extension of "organization". 

Do organizations have intelligence?  I think so.  Here's some of the reasons why:

1. We can model human organizations as having preference functions. (Economists do this all the time)
2. Human organizations have a lot of optimization power.

I talked with Mr. Muehlhauser about this specifically. I gather that at least at the time he thought human organizations should not be counted as intelligences (or at least as intelligences with the potential to become superintelligences) because they are not as versatile as human beings.

So when I am talking about super-human intelligence, I specifically mean an agent that is as good or better at humans at just about every skill set that humans possess for achieving their goals. So that would include things like not just mathematical ability or theorem proving and playing chess, but also things like social manipulation and composing music and so on, which are all functions of the brain not the kidneys

...and then...

It would be a kind of weird [organization] that was better than the best human or even the median human at all the things that humans do. [Organizations] aren’t usually the best in music and AI research and theory proving and stock markets and composing novels. And so there certainly are  [Organizations] that  are better than median humans at certain things, like digging oil wells, but I don’t think there are [Organizations] as good or better than humans at all things. More to the point, there is an interesting difference here because [Organizations] are made of lots of humans and so they have the sorts of limitations on activities and intelligence that humans have. For example, they are not particularly rational in the sense defined by cognitive science. And the brains of the people that make up organizations are limited to the size of skulls, whereas you can have an AI that is the size of a warehouse. 

I think that Muehlhauser is slightly mistaken on a few subtle but important points.  I'm going to assert my position on them without much argument because I think they are fairly sensible, but if any reader disagrees I will try to defend them in the comments.

• When judging whether an entity has intelligence, we should consider only the skills relevant to the entity's goals.
• So, if organizations are not as good at a human being at composing music, that shouldn't disqualify them from being considered broadly intelligent if that has nothing to do with their goals.
• Many organizations are quite good at AI research, or outsource their AI research to other organizations with which they are intertwined.
• The cognitive power of an organization is not limited to the size of skulls. The computational power is of many organizations is comprised of both the skulls of its members and possibly "warehouses" of digital computers.
• With the ubiquity of cloud computing, it's hard to say that a particular computational process has a static spatial bound at all.

Mean organizations

* My preferred standard of rationality is communicative rationality, a Habermasian ideal of a rationality aimed at consensus through principled communication.  As a consequence, when I believe a position to be rational, I believe that it is possible and desirable to convince other rational agents of it.

Article at io9. The paper is available here.

The researchers showed monkeys specific images and then trained them to select those images out of a larger set after a time delay. They recorded the monkeys' brain function to determine which signals were important. The experiment tests the monkey's performance on this task in different cases, as described by io9:

Once they were satisfied that the correct mapping had been done, they administered cocaine to the monkeys to impair their performance on the match-to-sample task (seems like a rather severe drug to administer, but there you have it). Immediately, the monkeys' performance fell by a factor of 20%.

It was at this point that the researchers engaged the neural device. Specifically, they deployed a "multi-input multi-output nonlinear" (MIMO) model to stimulate the neurons that the monkeys needed to complete the task. The inputs of this device monitored such things as blood flow, temperature, and the electrical activity of other neurons, while the outputs triggered the individual neurons required for decision making. Taken together, the i/o model was able to predict the output of the cortical neurons — and in turn deliver electrical stimulation to the right neurons at the right time.

And incredibly, it worked. The researchers successfully restored the monkeys' decision-making skills even though they were still dealing with the effects of the cocaine. Moreover, when duplicating the experiment under normal conditions, the monkeys' performance improved beyond the 75% proficiency level shown earlier. In other words, a kind of cognitive enhancement had happened.

This research is a remarkable followup to research that was done in rodents last year.

All the posts on FAI theory as of late have given me cause to think. There's something in the conversations about it that has always bugged me, but it is something that I haven't found the words for before now.

It is something like this:

Say that you manage to construct an algorithm for FAI...

Say that you can show that it isn't going to be a dangerous mistake...

And say you do all of this, and popularize it, before AGI is created (or at least, before an AGI goes *FOOM*)...

...

How in the name of Sagan are you actually going to ENFORCE the idea that all AGIs are FAIs? 

I mean, if it required some rare material (like nuclear weapons) or large laboratories (like biological wmds) or some other resource that you could at least make artificially scarce, you could set up a body that ensures that any AGI created is an FAI.

But if all it is, is the right algorithms, the right code, and enough computing power... even if you design a theory for FAI, how would you keep someone from making UFAI anyway? Between people experimenting with the principles (once known), making mistakes, and the prospect of actively malicious *humans*... it just seems like unless you somehow come up with an internal mechanism that makes FAI better and stronger than any UFAI could be, and the solution turns out to be such that any idiot could see that it was a better solution... that UFAI is going to exist at some point no matter what.

At that point, it seems like the question becomes not "How do we make FAI?" (although that might be a secondary question) but rather "How do we prevent the creation of, eliminate, or reduce potential damage from UFAI?" Now, it seems like FAI might be one thing that you do toward that goal, but if UFAI is a highly likely consequence of AGI even *with* an FAI theory, shouldn't the focus be on how to contain a UFAI event?

Restoring Memory, Repairing Damaged Brains (article @ PR Newswire)

Using an electronic system that duplicates the neural signals associated with memory, they managed to replicate the brain function in rats associated with long-term learned behavior, even when the rats had been drugged to forget.

This series of experiments, as described, sounds very well-constructed and thorough. The scientists first recorded specific activity in the hippocampus, where short-term memory becomes long-term memory. They then used drugs to inhibit that activity, preventing the formation of and access to long-term memory. Using the information they had gathered about the hippocampus activity, they constructed an artificial replacement and implanted it into the rats' brains. This successfully restored the rats' ability to store and use long-term memory. Further, they implanted the device into rats without suppressed hippocampal activity, and demonstrated increased memory abilities in those subjects.

"These integrated experimental modeling studies show for the first time that with sufficient information about the neural coding of memories, a neural prosthesis capable of real-time identification and manipulation of the encoding process can restore and even enhance cognitive mnemonic processes," says the paper.

It's a truly impressive result.

Ars Technica has an article about A Starcraft AI competition.. While this is clearly narrow AI there are some details which may interest people at LW. The article is about the best performing AI, the "Berkeley Overmind." (The AI in question only played as Zerg, one of the three possible sides in Starcraft. In fact, it seems that the AIs in general were all specialized for a single one of the three sides. While human players are often much better at one specific side, they are not nearly this specialized).

Highlights from the article:

StarCraft was released in 1998, an eternity ago by video game standards. Over those years Blizzard Entertainment, the game’s creator, has continually updated it so that it’s one of the most finely tuned and balanced Real Time Strategy (RTS) games ever made. It has three playable races: the human-like Terrans, with familiar tanks and starships, the alien Zerg, with large swarms of organic creatures, and the Protoss, technologically advanced aliens reliant on powerful but expensive units. Each race has different units and gameplay philosophies, yet no one race or combination of units has an unbeatable advantage. Player skill, ingenuity, and the ability to react intelligently to enemy actions determine victory.

This refinement and complexity makes StarCraft an ideal environment for conducting AI research. In an RTS game, events unfold in real-time and players’ orders are carried out immediately. Resources have to be gathered so fighting units can be produced and commanded into battle. The map is shrouded in fog-of-war, so enemy units and buildings are only visible when they’re near friendly buildings or units. A StarCraft player has to acquire and allocate resources to create units, coordinate those units in combat, discover, reason about and react to enemy actions, and do all this in real-time. These are all hard problems for a computer to solve.

Note, that using the interface that humans need to use was not one of the restrictions. This was an advantage that the Berkeley group used to full effect, as did other AIs in the comptetion.

We had to limit ourselves. David Burkett, another of Dan’s PhD students and the other team lead, says, “It turns out building control nodes for units is hard, so there’s a huge cost associated with building more than one [type of] unit. So we started asking: what one unit type [would be] the most effective overall?”

We focused our efforts on Zerg mutalisks: fast, dragon-like flying creatures that can attack both air and ground targets. Their mobility is unmatched, and we suspected they would be particularly amenable to computer control. Mutalisks are cheap for their strength, but large numbers are rarely seen in human play because it’s hard for a human to manage mutalisks without clumping them and making them easy prey for enemies with area attacks (attacks that do damage to all units in an area instead of a single target). A computer would have no such limitations.

The programmers then used a series of potential fields to control what the mutalisks did, with different entities and events creating different potential fields. A major issue became how to weigh these fields:

Using StarCraft’s map editor, we built Valhalla for the Overmind, where it could repeatedly and automatically run through different combat scenarios. By running repeated trials in Valhalla and varying the potential field strengths, the agent learned the best combination of parameters for each kind of engagement.

The article unfortunately doesn't go into great detail about the exact learning mechanism. Note however that this implies that the Overmind should be able to learn how to respond to other unit types.

There are other details in the article that are also interesting. For example, they replaced the standard path tracing algorithm that units do automatically with their own algorithms.

The final form of the AI can play well against very skilled human players, but it isn't at the top of the game. Note also that the Overmind is designed for one-on-one games. It should be interesting to see how this AI and similar AIs improve over the next few years. I'd be very curious how an AIXI would do in this sort of situation.

There's a good song by Eminem - If I had a million dollars.  So, if I had a hypothetical task to give away $30 million to different foundations without having a right to influence the projects, I would distribute them as follows, $3 million for each organization:

1. Nanofactory collaboration, Robert Freitas, Ralph Merkle – developers of molecular nanotechnology and nanomedicine. Robert Freitas is the author of the monography Nanomedicine.
2. Singularity institute, Michael Vassar, Eliezer Yudkowsky – developers and ideologists of the friendly Artificial Intelligence
3. SENS Foundation, Aubrey de Grey – the most active engineering project in life extension, focused on the most promising underfunded areas
4. Cryonics Institute – one of the biggest cryonics firms in the US, they are able to use the additional funding more effectively as compared to Alcor
5. Advanced Neural Biosciences, Aschwin de Wolf – an independent cryonics research center created by ex-researchers from Suspended Animation
6. Brain observatory – brain scanning
7. University Hospital Careggi in Florence, Paolo Macchiarini – growing organs (not an American medical school, because this amount of money won’t make any difference to the leading American centers)
8. Immortality institute – advocating for immortalism, selected experiments
9. IEET – institute of ethics and emerging technologies – promotion of transhumanist ideas
10. Small research grants of $50-300 thousand

Now, if the task is to most effectively invest $30 million dollars, what projects would be chosen? (By effectiveness here I mean increasing the chances of radical life extension)

Well, off the top of my head:

1. The project: “Creation of technologies to grow a human liver” – $7 million. The project itself costs approximately $30-50 million, but $7 million is enough to achieve some significant intermediate results and will definitely attract more funds from potential investors.
2. Break the world record in sustaining viability of a mammalian head separate from the body - $0.7 million
3. Creation of an information system, which characterizes data on changes during aging in humans, integrates biomarkers of aging, and evaluates the role of pharmacological and other interventions in aging processes – $3 million
4. Research in increasing cryoprotectors efficacy - $3 million
5. Creation and realization of a program “Regulation of epigenome” - $5 million
6. Creation, promotion and lobbying of the program on research and fighting aging - $2 million
7. Educational programs in the fields of biogerontology, neuromodelling, regenerative medicine, engineered organs - $1.5 million
8. “Artificial blood” project - $2 million
9. Grants for authors, script writers, and art representatives for creation of pieces promoting transhumanism - $0.5 million
10. SENS Foundation project of removing senescent cells - $2 million
11. Creation of a US-based non-profit, which would protect and lobby the right to live and scientific research in life extension - $2 million
11. Participation of  “H+ managers” in conferences, forums  and social events - $1 million
12. Advocacy and creating content in social media - $0.3 million

In the novel Life Artificial I use the following assumptions regarding the creation and employment of AI personalities.

1. AI is too complex to be designed; instances are evolved in batches, with successful ones reproduced
2. After an initial training period, the AI must earn its keep by paying for Time (a unit of computational use)

(Note: PDA := AI, Sticky := human)

The second fitness gradient is based on economics and social considerations: can an AI actually earn a living? Otherwise it gets turned off.

As a result of following this line of thinking, it seems obvious that after the initial novelty wears off, AIs will be terribly mistreated (anthropomorphizing, yeah).

It would be very forward-thinking to begin to engineer barriers to such mistreatment, like a PETA for AIs. It is interesting that such an organization already exists, at least on the Internet: ASPCR