Articles Tagged ‘ethics’ - LessWrong

A brief history of ethically concerned scientists

Sat, 09 Feb 2013 16:50:00 +1100

Submitted by Kaj_Sotala • 67 votes • 146 comments

For the first time in history, it has become possible for a limited group of a few thousand people to threaten the absolute destruction of millions.

-- Norbert Wiener (1956), Moral Reflections of a Mathematician.

Today, the general attitude towards scientific discovery is that scientists are not themselves responsible for how their work is used. For someone who is interested in science for its own sake, or even for someone who mostly considers research to be a way to pay the bills, this is a tempting attitude. It would be easy to only focus on one’s work, and leave it up to others to decide what to do with it.

But this is not necessarily the attitude that we should encourage. As technology becomes more powerful, it also becomes more dangerous. Throughout history, many scientists and inventors have recognized this, and taken different kinds of action to help ensure that their work will have beneficial consequences. Here are some of them.

This post is not arguing that any specific approach for taking responsibility for one's actions is the correct one. Some researchers hid their work, others refocused on other fields, still others began active campaigns to change the way their work was being used. It is up to the reader to decide which of these approaches were successful and worth emulating, and which ones were not.

Pre-industrial inventors

… I do not publish nor divulge [methods of building submarines] by reason of the evil nature of men who would use them as means of destruction at the bottom of the sea, by sending ships to the bottom, and sinking them together with the men in them.

-- Leonardo da Vinci

People did not always think that the benefits of freely disseminating knowledge outweighed the harms. O.T. Benfey, writing in a 1956 issue of the Bulletin of the Atomic Scientists, cites F.S. Taylor’s book on early alchemists:

Alchemy was certainly intended to be useful .... But [the alchemist] never proposes the public use of such things, the disclosing of his knowledge for the benefit of man. …. Any disclosure of the alchemical secret was felt to be profoundly wrong, and likely to bring immediate punishment from on high. The reason generally given for such secrecy was the probable abuse by wicked men of the power that the alchemical would give …. The alchemists, indeed, felt a strong moral responsibility that is not always acknowledged by the scientists of today.

With the Renaissance, science began to be viewed as public property, but many scientists remained cautious about the way in which their work might be used. Although he held the office of military engineer, Leonardo da Vinci (1452-1519) drew a distinction between offensive and defensive warfare, and emphasized the role of good defenses in protecting people’s liberty from tyrants. He described war as ‘bestialissima pazzia’ (most bestial madness), and wrote that ‘it is an infinitely atrocious thing to take away the life of a man’. One of the clearest examples of his reluctance to unleash dangerous inventions was his refusal to publish the details of his plans for submarines.

Later Renaissance thinkers continued to be concerned with the potential uses of their discoveries. John Napier (1550-1617), the inventor of logarithms, also experimented with a new form of artillery. Upon seeing its destructive power, he decided to keep its details a secret, and even spoke from his deathbed against the creation of new kinds of weapons.

But only concealing one discovery pales in comparison to the likes of Robert Boyle (1627-1691). A pioneer of physics and chemistry and possibly the most famous for describing and publishing Boyle’s law, he sought to make humanity better off, taking an interest in things such as improved agricultural methods as well as better medicine. In his studies, he also discovered knowledge and made inventions related to a variety of potentially harmful subjects, including poisons, invisible ink, counterfeit money, explosives, and kinetic weaponry. These ‘my love of Mankind has oblig’d me to conceal, even from my nearest Friends’.

Chemical warfare

By the early twentieth century, people had began looking at science in an increasingly optimistic light: it was believed that science would not only continue to improve everyone’s prosperity, but also make war outright impossible. But as science became more sophisticated, it would also become possible to cause ever more harm with ever smaller resources. One of the early indications of science’s ability to do harm came from advances in chemical warfare, and World War I saw the deployment of chlorine, phosgene, and mustard gas as weapons. It should not be surprising, then, that some scientists in related fields began growing concerned. But unlike earlier inventors, at least three of them did far more than just refuse to publish their work.

Clara Immerwahr (1870-1915) was a German chemist and the first woman to obtain a PhD from the University of Breslau. She was strongly opposed to the use of chemical weapons. Married to Fritz Haber, ‘the father of chemical warfare’, she unsuccessfully attempted many times to convince her husband to abandon his work. Immerwahr was generally depressed and miserable over the fact that society considered a married woman’s place to be at home, denying her the opportunity to do science. In the end, after her efforts to dissuade her husband from working on chemical warfare had failed and Fritz had personally overseen the first major use of chlorine, she committed suicide by shooting herself in the heart.

Poison gas also concerned scientists in other disciplines. Lewis Fry Richardson (1881-1953) was a mathematician and meteorologist. During the World War II, the military became interested in his work on turbulence and gas mixing, and attempted to recruit him to do help them do work on modeling the best ways of using poison gas. Realizing what his work was being used for, Richardson abandoned meteorology entirely and destroyed his unpublished research. Instead, he turned his research to investigating the causes of war, attempting to find ways to reduce the risk of armed conflict. He spent the rest of his life devoted to this topic, and is today considered one of the founders of the scientific analysis of conflict.

Arthur Galston (1920-2008), a botanist, was also concerned with the military use of his inventions. Building upon his work, the US military developed Agent Orange, a chemical weapon which was deployed in the Vietnam War. Upon discovering what his work had been used for, he began to campaign against its use, and together with a number of others finally convinced President Nixon to order an end to its spraying in 1970. Reflecting upon the matter, Galston wrote:

I used to think that one could avoid involvement in the antisocial consequences of science simply by not working on any project that might be turned to evil or destructive ends. I have learned that things are not all that simple, and that almost any scientific finding can be perverted or twisted under appropriate societal pressures. In my view, the only recourse for a scientist concerned about the social consequences of his work is to remain involved with it to the end. His responsibility to society does not cease with publication of a definitive scientific paper. Rather, if his discovery is translated into some impact on the world outside the laboratory, he will, in most instances, want to follow through to see that it is used for constructive rather than anti-human purposes.

After retiring in 1990, he founded the Interdisciplinary Center for Bioethics at Yale, where he also taught bioethics to undergraduates.

Nuclear weapons

While chemical weapons are capable of inflicting serious injuries as well as birth defects on large numbers of people, they have never been viewed to be as dangerous as nuclear weapons. As physicists became capable of creating weapons of unparalleled destructive power, they also began growing ever more concerned about the consequences of their work.

Leó Szilárd (1898-1964) was one of the first people to envision nuclear weapons, and was granted a patent for the nuclear chain reaction in 1934. Two years later, he grew worried that Nazi scientists would find his patents and use them to create weapons, so he asked the British Patent Office to withdraw his patents and secretly reassign them to the Royal Navy. His fear of Nazi Germany developing nuclear weapons also made him instrumental in making the USA initiate the Manhattan Project, as he and two other scientists wrote the Einstein-Szilárd letter that advised President Roosevelt of the need to develop the same technology. But in 1945, he learned that the atomic bomb was about to be used on Japan, despite it being certain that neither Germany nor Japan had one. He then did his best to stop them from being used and started a petition against using them, with little success.

After the war, he no longer wanted to contribute to the creation of weapons and changed fields to molecular biology. In 1962, he founded the Council for a Livable World, which aimed to warn people about the dangers of nuclear war and to promote a policy of arms control. The Council continues its work even today.

Another physicist who worked on the atomic bomb due to a fear of it being developed by Nazi Germany was Joseph Rotblat (1908-2005), who felt that the Allies also having an atomic bomb would deter the Axis from using one. But he gradually began to realize that Nazi Germany would likely never develop the atomic bomb, destroying his initial argument for working on it. He also came to realize that the bomb continued to be under active development due to reasons that he felt were unethical. In conversation, General Leslie Groves mentioned that the real purpose of the bomb was to subdue the USSR. Rotblat was shocked to hear this, especially given that the Soviet Union was at the time an ally in the war effort. In 1944, it became apparent that Germany would not develop the atomic bomb. As a result, Rotblat asked for permission to leave the project, and was granted it.

Afterwards, Rotblat regretted his role in developing nuclear weapons. He believed that the logic of nuclear deterrence was flawed, since he thought that if Hitler had possessed an atomic bomb, then Hitler’s last order would have been to use it against London regardless of the consequences. Rotblat decided to do whatever he could to prevent the future use and deployment of nuclear weapons, and proposed a worldwide moratorium on such research until humanity was wise enough to use it without risks. He decided to repurpose his career into something more useful for humanity, and began studying and teaching the application of nuclear physics into medicine, becoming a professor at the Medical College of St Bartholomew’s Hospital in London.

Rotblat worked together with Bertrand Russell to limit the spread of nuclear weapons, and the two collaborated with a number of other scientists to issue the Russell-Einstein Manifesto in 1955, calling the governments of the world to take action to prevent nuclear weapons from doing more damage. The manifesto led to the establishment of the Pugwash Conferences, in which nuclear scientists from both the West and the East met each other. By facilitating dialogue between the two sides of the Cold War, these conferences helped lead to several arms control agreements, such as the Partial Test Ban Treaty of 1963 and the Non-Proliferation Treaty of 1968. In 1995, Rotblat and the Pugwash Conferences were awarded the Nobel Peace Prize “for their efforts to diminish the part played by nuclear arms in international politics and, in the longer run, to eliminate such arms”.

The development of nuclear weapons also affected Norbert Wiener (1894-1964), professor of mathematics at the Massachusetts Institute of Technology and the originator of the field of cybernetics. After the Hiroshima bombing, a researcher working for a major aircraft corporation requested a copy of an earlier paper of Wiener’s. Wiener refused to provide it, and sent Atlantic Monthly a copy of his response to the researcher, in which he declared his refusal to share his research with anyone who would use it for military purposes.

In the past, the community of scholars has made it a custom to furnish scientific information to any person seriously seeking it. However, we must face these facts: The policy of the government itself during and after the war, say in the bombing of Hiroshima and Nagasaki, has made it clear that to provide scientific information is not a necessarily innocent act, and may entail the gravest consequences. One therefore cannot escape reconsidering the established custom of the scientist to give information to every person who may inquire of him. The interchange of ideas, one of the great traditions of science, must of course receive certain limitations when the scientist becomes an arbiter of life and death. [...]

The experience of the scientists who have worked on the atomic bomb has indicated that in any investigation of this kind the scientist ends by putting unlimited powers in the hands of the people whom he is least inclined to trust with their use. It is perfectly clear also that to disseminate information about a weapon in the present state of our civilization is to make it practically certain that that weapon will be used. [...]

If therefore I do not desire to participate in the bombing or poisoning of defenseless peoples-and I most certainly do not-I must take a serious responsibility as to those to whom I disclose my scientific ideas. Since it is obvious that with sufficient effort you can obtain my material, even though it is out of print, I can only protest pro forma in refusing to give you any information concerning my past work. However, I rejoice at the fact that my material is not readily available, inasmuch as it gives me the opportunity to raise this serious moral issue. I do not expect to publish any future work of mine which may do damage in the hands of irresponsible militarists.

I am taking the liberty of calling this letter to the attention of other people in scientific work. I believe it is only proper that they should know of it in order to make their own independent decisions, if similar situations should confront them.

Recombinant DNA

For a large part of history, scientists’ largest ethical concerns came from direct military applications of their inventions. While any invention could lead to unintended societal or environmental consequences, for the most part researchers who worked on peaceful technologies didn’t need to be too concerned with their work being dangerous by itself. But as biological and medical research obtained the capability to modify genes and bacteria, it would open up the possibility of unintentionally creating dangerous infectious diseases. In theory, these could be even more dangerous than nuclear weapons - an a-bomb dropped on a city might destroy most of that city, but a single bacteria could give rise to an epidemic infecting people all around the world.

Recombinant DNA techniques involve taking DNA from one source and then introducing it to another kind of organism, causing the new genes to express themselves in the target organism. One of the pioneers of this technique was Paul Berg (1926-), who in 1972 had already carried out the preparations for creating a strain of E. coli that contained the genome for a human-infectious virus (SV40) with tentative links to cancer. Robert Pollack (1920-) heard news of this experiment and helped convince Berg to halt it - both were concerned about the danger that this new strain would spread to humans in the lab and become a pathogen. Berg then became a major voice calling for more attention to the risks of such research as well as a temporary moratorium. This eventually led to two conferences in Asilomar, with 140 experts participating in the later 1975 one to decide upon guidelines for recombinant DNA research.

Berg and Pollack were far from the only scientists to call attention to the safety concerns of recombinant DNA. Several other scientists contributed, asking for more safety and voicing concern about a technology that could bring harm if misused.

Among them, the molecular biologist Maxine Singer (1931-) chaired the 1973 Gordon Conference on Nucleic Acids, in which some of the dangers of the technique were discussed. After the conference, she and several other similarly concerned scientists authored a letter to the President of the National Academy of Science and the President of the Institutes of Health. The letter suggested that a study committee be established to study the risks behind the new recombinant DNA technology, and propose specific actions or guidelines if necessary. She also helped organize the Asilomar Conference in 1975.

Informatics

But if we are downloaded into our technology, what are the chances that we will thereafter be ourselves or even human? It seems to me far more likely that a robotic existence would not be like a human one in any sense that we understand, that the robots would in no sense be our children, that on this path our humanity may well be lost.

-- Bill Joy, Why the Future Doesn’t Need Us.

Finally, we come to the topic of information technology and artificial intelligence. As AI systems grow increasingly autonomous, they might become the ultimate example of a technology that seems initially innocuous but ends up capable of doing great damage. Especially if they were to become capable of rapid self-improvement, they could lead to humanity going extinct.

In addition to refusing to help military research, Norbert Wiener was also concerned about the effects of automation. In 1949, General Electric wanted him to advise its managers on automaton matters and to teach automation methods to its engineers. Wiener refused these requests, believing that they would further a development which would lead to human workers becoming unemployed and replaced by machines. He thus expanded his boycott of the military to also be a boycott of corporations that he thought acted unethically.

Wiener was also concerned about the risks of autonomous AI. In 1960, Science published his paper "Some Moral and Technical Consequences of Automation", in which he spoke at length about the dangers of machine intelligence. He warned that machines might act far too fast for humans to correct their mistakes, and that like genies in stories, they could fulfill the letter of our requests without caring about their spirit. He also discussed such worries elsewhere.

If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.

Such worries would continue to bother other computer scientists as well, many decades after Wiener’s death. Bill Joy (1954-) is known for having played a major role in the development of BSD Unix, having authored the vi text editor, and being the co-founder of Sun Microsystems. He became concerned about the effects of AI in 1998, when he met Ray Kurzweil at a conference where they were both speakers. Kurzweil gave Joy a preprint of his then-upcoming book, The Age of Spiritual Machines, and Joy found himself concerned over its discussion about the risks of AI. Reading Hans Moravec’s book Robot: Mere Machine to Transcendent Mind exacerbated Joy’s worries, as did several other books which he found around the same time. He began to wonder whether all of his work in the field of information technology and computing had been preparing the way for a world where machines would replace humans.

In 2000, Joy wrote a widely-read article titled Why the Future Doesn’t Need Us for Wired, talking about the dangers of AI as well as genetic engineering and nanotechnology. In the article, he called to limit the development of technologies which he felt were too dangerous. Since then, he has continued to be active in promoting responsible technology research. In 2005, an op-ed co-authored by Joy and Ray Kurzweil was published in the New York Times, arguing that the decision to publish the genome of the 1918 influenza virus on the Internet had been a mistake.

Joy also attempted to write a book on the topic, but then became convinced that he could achieve more by working on science and technology investment. In 2005, he joined the venture capital firm Kleiner Perkins Caufield & Byers as a partner, and he has been focused on investments in green technology.

Conclusion

Technology's potential for destruction will only continue to grow, but many of the social norms of science were established under the assumption that scientists don’t need to worry much about how the results of their work are used. Hopefully, the examples provided in this post can encourage more researchers to consider the broader consequences of their work.

Sources used

This article was written based on research done by Vincent Fagot. The sources listed below are in addition to any that are already linked from the text.

Leonardo da Vinci:

“The Notebooks of Leonardo da Vinci” vol 1, by Edward Mac Curdy (1905 edition)
“The scientist’s conscience : historical considerations” in Bulletin of the Atomic Scientists - May 1956 - Page 177

John Napier:

Robert Boyle:

Clara Immerwahr:

Lewis Fry Richardson:

T. W. Körner : The Pleasures of Counting

“The scientist’s conscience : historical considerations” in Bulletin of the Atomic Scientists - May 1956 - Page 177

Arthur Galston:

Leó Szilárd:

Joseph Rotblat:

Bulletin of the atomic scientists, august 1985 : Leaving the Bomb Project by Joseph Rotblat
Keeper of the Nuclear Conscience: The Life and Work of Joseph Rotblat
1999 voice record interview of Joseph Rotblat
Deriving an Ethical Code for Scientists: An Interview With Joseph Rotblat

Norbert Wiener:

Postmodern War: The New Politics of Conflict by Chris Hables Gray (available online)
Bulletin of the Atomic Scientists - May 1956 - Page 178 : “The scientist’s conscience : historical considerations
Dark Hero of the Information Age: In Search of Norbert Wiener The Father of Cybernetics by Flo Conway
Bulletin of the Atomic Scientists - Jan 1947 - Page 31 : "A Scientist Rebels"
Bulletin of the Atomic Scientists - Feb 1956 - Page 53 : "Moral Reflections of a Mathematician"
Bulletin of the Atomic Scientists - Nov 1948 - Page 338 : "A Rebellious Scientist After Two Years"
Rosemary Chalk : Drawing the Line An Examination of Conscientious Objection in Science
Some Moral and Technical Consequences of Automation in Science, 6 may 1960

Paul Berg, Maxine Singer, Robert Pollack:

P. Berg and M. F. Singer : The recombinant DNA controversy: twenty years later, Proc Natl Acad Sci U S A. 1995 September 26
Potential Biohazards of Recombinant DNA Molecules by Paul Berg, 1974
Guidelines for DNA Hybrid Molecules by Maxine Singer, 1973
Biomedical Politics by Kathi E. Hanna (chapter : Asilomar and recombinant DNA by Donald S. Fredrickson), 1991
Asilomar Conference on Laboratory Precautions When Conducting Recombinant DNA Research – Case Summary
Report - Assembly of Life Sciences, National Research Council
P. Berg: Potential Biohazards of Recombinant DNA Molecules
Watson and DNA: Making a Scientific Revolution

146 comments

Holden's Objection 1: Friendliness is dangerous

Fri, 18 May 2012 10:48:14 +1000

Submitted by PhilGoetz • 11 votes • 428 comments

Nick_Beckstead asked me to link to posts I referred to in this comment. I should put up or shut up, so here's an attempt to give an organized overview of them.

Since I wrote these, LukeProg has begun tackling some related issues. He has accomplished the seemingly-impossible task of writing many long, substantive posts none of which I recall disagreeing with. And I have, irrationally, not read most of his posts. So he may have dealt with more of these same issues.

I think that I only raised Holden's "objection 2" in comments, which I couldn't easily dig up; and in a critique of a book chapter, which I emailed to LukeProg and did not post to LessWrong. So I'm only going to talk about "Objection 1: It seems to me that any AGI that was set to maximize a "Friendly" utility function would be extraordinarily dangerous." I've arranged my previous posts and comments on this point into categories. (Much of what I've said on the topic has been in comments on LessWrong and Overcoming Bias, and in email lists including SL4, and isn't here.)

The concept of "human values" cannot be defined in the way that FAI presupposes

Human errors, human values: Suppose all humans shared an identical set of values, preferences, and biases. We cannot retain human values without retaining human errors, because there is no principled distinction between them.

A comment on this post: There are at least three distinct levels of human values: The values an evolutionary agent holds that maximize their reproductive fitness, the values a society holds that maximizes its fitness, and the values a rational optimizer holds who has chosen to maximize social utility. They often conflict. Which of them are the real human values?

Values vs. parameters: Eliezer has suggested using human values, but without time discounting (= changing the time-discounting parameter). CEV presupposes that we can abstract human values and apply them in a different situation that has different parameters. But the parameters are values. There is no distinction between parameters and values.

A comment on "Incremental progress and the valley": The "values" that our brains try to maximize in the short run are designed to maximize different values for our bodies in the long run. Which are human values: The motivations we feel, or the effects they have in the long term? LukeProg's post Do Humans Want Things? makes a related point.

Group selection update: The reason I harp on group selection, besides my outrage at the way it's been treated for the past 50 years, is that group selection implies that some human values evolved at the group level, not at the level of the individual. This means that increasing the rationality of individuals may enable people to act more effectively in their own interests, rather than in the group's interest, and thus diminish the degree to which humans embody human values. Identifying the values embodied in individual humans - supposing we could do so - would still not arrive at human values. Transferring human values to a post-human world, which might contain groups at many different levels of a hierarchy, would be problematic.

I wanted to write about my opinion that human values can't be divided into final values and instrumental values, the way discussion of FAI presumes they can. This is an idea that comes from mathematics, symbolic logic, and classical AI. A symbolic approach would probably make proving safety easier. But human brains don't work that way. You can and do change your values over time, because you don't really have terminal values.

Strictly speaking, it is impossible for an agent whose goals are all indexical goals describing states involving itself to have preferences about a situation in which it does not exist. Those of you who are operating under the assumption that we are maximizing a utility function with evolved terminal goals, should I think admit these terminal goals all involve either ourselves, or our genes. If they involve ourselves, then utility functions based on these goals cannot even be computed once we die. If they involve our genes, they they are goals that our bodies are pursuing, that we call errors, not goals, when we the conscious agent inside our bodies evaluate them. In either case, there is no logical reason for us to wish to maximize some utility function based on these after our own deaths. Any action I wish to take regarding the distant future necessarily presupposes that the entire SIAI approach to goals is wrong.

My view, under which it does make sense for me to say I have preferences about the distant future, is that my mind has learned "values" that are not symbols, but analog numbers distributed among neurons. As described in "Only humans can have human values", these values do not exist in a hierarchy with some at the bottom and some on the top, but in a recurrent network which does not have a top or a bottom, because the different parts of the network developed simultaneously. These values therefore can't be categorized into instrumental or terminal. They can include very abstract values that don't need to refer specifically to me, because other values elsewhere in the network do refer to me, and this will ensure that actions I finally execute incorporating those values are also influenced by my other values that do talk about me.

Even if human values existed, it would be pointless to preserve them

Only humans can have human values:

The only preferences that can be unambiguously determined are the preferences a person (mind+body) implements, which are not always the preferences expressed by their beliefs.
If you extract a set of consciously-believed propositions from an existing agent, then build a new agent to use those propositions in a different environment, with an "improved" logic, you can't claim that it has the same values, since it will behave differently.
Values exist in a network of other values. A key ethical question is to what degree values are referential (meaning they can be tested against something outside that network); or non-referential (and hence relative).
Supposing that values are referential helps only by telling you to ignore human values.
You cannot resolve the problem by combining information from different behaviors, because the needed information is missing.
Today's ethical disagreements are largely the result of attempting to extrapolate ancestral human values into a changing world.
The future will thus be ethically contentious even if we accurately characterize and agree on present human values, because these values will fail to address the new important problems.

Human values differ as much as values can differ: There are two fundamentally different categories of values:

Non-positional, mutually-satisfiable values (physical luxury, for instance)
Positional, zero-sum social values, such as wanting to be the alpha male or the homecoming queen

All mutually-satisfiable values have more in common with each other than they do with any non-mutually-satisfiable values, because mutually-satisfiable values are compatible with social harmony and non-problematic utility maximization, while non- mutually-satisfiable values require eternal conflict. If you find an alien life form from a distant galaxy with non-positional values, it would be easier to integrate those values into a human culture with only human non-positional values, than to integrate already-existing positional human values into that culture.

It appears that some humans have mainly the one type, while other humans have mainly the other type. So talking about trying to preserve human values is pointless - the values held by different humans have already passed the most-important point of divergence.

Enforcing human values would be harmful

The human problem: This argues that the qualia and values we have now are only the beginning of those that could evolve in the universe, and that ensuring that we maximize human values - or any existing value set - from now on, will stop this process in its tracks, and prevent anything better from ever evolving. This is the most-important objection of all.

Re-reading this, I see that the critical paragraph is painfully obscure, as if written by Kant; but it summarizes the argument: "Once the initial symbol set has been chosen, the semantics must be set in stone for the judging function to be "safe" for preserving value; this means that any new symbols must be defined completely in terms of already-existing symbols. Because fine-grained sensory information has been lost, new developments in consciousness might not be detectable in the symbolic representation after the abstraction process. If they are detectable via statistical correlations between existing concepts, they will be difficult to reify parsimoniously as a composite of existing symbols. Not using a theory of phenomenology means that no effort is being made to look for such new developments, making their detection and reification even more unlikely. And an evaluation based on already-developed values and qualia means that even if they could be found, new ones would not improve the score. Competition for high scores on the existing function, plus lack of selection for components orthogonal to that function, will ensure that no such new developments last."

Averaging value systems is worse than choosing one: This describes a neural-network that encodes preferences, and takes some input pattern and computes a new pattern that optimizes these preferences. Such a system is taken as analogous for a value system and an ethical system to attain those values. I then define a measure for the internal conflict produced by a set of values, and show that a system built by averaging together the parameters from many different systems will have higher internal conflict than any of the systems that were averaged together to produce it. The point is that the CEV plan of "averaging together" human values will result in a set of values that is worse (more self-contradictory) than any of the value systems it was derived from.

A point I may not have made in these posts, but made in comments, is that the majority of humans today think that women should not have full rights, homosexuals should be killed or at least severely persecuted, and nerds should be given wedgies. These are not incompletely-extrapolated values that will change with more information; they are values. Opponents of gay marriage make it clear that they do not object to gay marriage based on a long-range utilitarian calculation; they directly value not allowing gays to marry. Many human values horrify most people on this list, so they shouldn't be trying to preserve them.

428 comments

Objections to Coherent Extrapolated Volition

Tue, 22 Nov 2011 21:32:13 +1100

Submitted by XiXiDu • 11 votes • 52 comments

In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.

— Eliezer Yudkowsky, May 2004, Coherent Extrapolated Volition

Foragers versus industry era folks

Consider the difference between a hunter-gatherer, who cares about his hunting success and to become the new tribal chief, and a modern computer scientist who wants to determine if a “sufficiently large randomized Conway board could turn out to converge to a barren ‘all off’ state.”

The utility of the success in hunting down animals and proving abstract conjectures about cellular automata is largely determined by factors such as your education, culture and environmental circumstances. The same forager who cared to kill a lot of animals, to get the best ladies in its clan, might have under different circumstances turned out to be a vegetarian mathematician solely caring about his understanding of the nature of reality. Both sets of values are to some extent mutually exclusive or at least disjoint. Yet both sets of values are what the person wants, given the circumstances. Change the circumstances dramatically and you change the persons values.

What do you really want?

You might conclude that what the hunter-gatherer really wants is to solve abstract mathematical problems, he just doesn’t know it. But there is no set of values that a person “really” wants. Humans are largely defined by the circumstances they reside in.

If you already knew a movie, you wouldn’t watch it.
To be able to get your meat from the supermarket changes the value of hunting.

If “we knew more, thought faster, were more the people we wished we were, and had grown up closer together” then we would stop to desire what we learnt, wish to think even faster, become even different people and get bored of and rise up from the people similar to us.

A singleton is an attractor

A singleton will inevitably change everything by causing a feedback loop between itself as an attractor and humans and their values.

Much of our values and goals, what we want, are culturally induced or the result of our ignorance. Reduce our ignorance and you change our values. One trivial example is our intellectual curiosity. If we don’t need to figure out what we want on our own, our curiosity is impaired.

A singleton won’t extrapolate human volition but implement an artificial set values as a result of abstract high-order contemplations about rational conduct.

With knowledge comes responsibility, with wisdom comes sorrow

Knowledge changes and introduces terminal goals. The toolkit that is called ‘rationality’, the rules and heuristics developed to help us to achieve our terminal goals are also altering and deleting them. A stone age hunter-gatherer seems to possess very different values than we do. Learning about rationality and various ethical theories such as Utilitarianism would alter those values considerably.

Rationality was meant to help us achieve our goals, e.g. become a better hunter. Rationality was designed to tell us what we ought to do (instrumental goals) to achieve what we want to do (terminal goals). Yet what actually happens is that we are told, that we will learn, what we ought to want.

If an agent becomes more knowledgeable and smarter then this does not leave its goal-reward-system intact if it is not especially designed to be stable. An agent who originally wanted to become a better hunter and feed his tribe would end up wanting to eliminate poverty in Obscureistan. The question is, how much of this new “wanting” is the result of using rationality to achieve terminal goals and how much is a side-effect of using rationality, how much is left of the original values versus the values induced by a feedback loop between the toolkit and its user?

Take for example an agent that is facing the Prisoner’s dilemma. Such an agent might originally tend to cooperate and only after learning about game theory decide to defect and gain a greater payoff. Was it rational for the agent to learn about game theory, in the sense that it helped the agent to achieve its goal or in the sense that it deleted one of its goals in exchange for a allegedly more “valuable” goal?

Beware rationality as a purpose in and of itself

It seems to me that becoming more knowledgeable and smarter is gradually altering our utility functions. But what is it that we are approaching if the extrapolation of our volition becomes a purpose in and of itself? Extrapolating our coherent volition will distort or alter what we really value by installing a new cognitive toolkit designed to achieve an equilibrium between us and other agents with the same toolkit.

Would a singleton be a tool that we can use to get what we want or would the tool use us to do what it does, would we be modeled or would it create models, would we be extrapolating our volition or rather follow our extrapolations?

_{(This post is a write-up of a previous comment designated to receive feedback from a larger audience.)}

52 comments

Morality is not about willpower

Sat, 08 Oct 2011 12:33:09 +1100

Submitted by PhilGoetz • 9 votes • 144 comments

Most people believe the way to lose weight is through willpower. My successful experience losing weight is that this is not the case. You will lose weight if you want to, meaning you effectively believe⁰ that the utility you will gain from losing weight, even time-discounted, will outweigh the utility from yummy food now. In LW terms, you will lose weight if your utility function tells you to. This is the basis of cognitive behavioral therapy (the effective kind of therapy), which tries to change peoples' behavior by examining their beliefs and changing their thinking habits.

Similarly, most people believe behaving ethically is a matter of willpower; and I believe this even less. Your ethics is part of your utility function. Acting morally is, technically, a choice; but not the difficult kind that holds up a stop sign and says "Choose wisely!" We notice difficult moral choices more than easy moral choices; but most moral choices are easy, like choosing a ten dollar bill over a five. Immorality is not a continual temptation we must resist; it's just a kind of stupidity.

This post can be summarized as:

Each normal human has an instinctive personal morality.
This morality consists of inputs into that human's decision-making system. There is no need to propose separate moral and selfish decision-making systems.
Acknowledging that all decisions are made by a single decision-making system, and that the moral elements enter it in the same manner as other preferences, results in many changes to how we encourage social behavior.

Many people have commented that humans don't make decisions based on utility functions. This is a surprising attitude to find on LessWrong, given that Eliezer has often cast rationality and moral reasoning in terms of computing expected utility. It also demonstrates a misunderstanding of what utility functions are. Values, and utility functions, are models we construct to explain why we do what we do. You can construct a set of values and a utility function to fit your observed behavior, no matter how your brain produces that behavior. You can fit this model to the data arbitrarily well by adding parameters. It will always have some error, as you are running on stochastic hardware. Behavior is not a product of the utility function; the utility function is a product of (and predictor of) the behavior. If your behavior can't be modelled with values and a utility function, you shouldn't bother reading LessWrong, because "being less wrong" means behaving in a way that is closer to the predictions of some model of rationality. If you are a mysterious black box with inscrutable motives that makes unpredictable actions, no one can say you are "wrong" about anything.

If you still insist that I shouldn't talk about utility functions, though - it doesn't matter! This post is about morality, not about utility functions. I use utility functions just as a way of saying "what you want to do". Substitute your own model of behavior. The bottom line here is that moral behavior is not a qualitatively separate type of behavior and does not require a separate model of behavior.

My view isn't new. It derives from ancient Greek ethics, Nietzsche, Ayn Rand, B.F. Skinner, and comments on LessWrong. I thought it was the dominant view on LW, but the comments and votes indicate it is held at best by a weak majority.

Relevant EY posts include "What would you do without morality?", "The gift we give to tomorrow", "Changing your meta-ethics", and "The meaning of right"; and particularly the statement, "Maybe that which you would do even if there were no morality, is your morality." I was surprised that no comments mentioned any of the many points of contact between this post and Eliezer's longest sequence. (Did anyone even read the entire meta-ethics sequence?) The view I'm presenting is, as far as I can tell, the same as that given in EY's meta-ethics sequence up through "The meaning of right"¹; but I am talking about what it is that people are doing when they act in a way we recognize as ethical, whereas Eliezer was talking about where people get their notions of what is ethical.

Ethics as willpower

Society's main story is that behaving morally means constantly making tough decisions and doing things you don't want to do. You have desires; other people have other desires; and ethics is a referee that helps us mutually satisfy these desires, or at least not kill each other. There is one true ethics; society tries to discover and encode it; and the moral choice is to follow that code.

This story has implications that usually go together:

Ethics is about when peoples' desires conflict. Thus, ethics is only concerned with interpersonal relations.
There is a single, Platonic, correct ethical system for a given X. (X used be a social class but not a context or society. Nowadays it can be a society or context but not a social class.)
Your desires and feelings are anti-correlated with ethical behavior. Humans are naturally unethical. Being ethical is a continual, lifelong struggle.
The main purpose of ethics is to stop people from doing what they naturally want to do, so "thou shalt not" is more important than "thou shalt".
The key to being ethical is having the willpower not to follow your own utility function.
Social ethics are encouraged by teaching people to "be good", where "good" is the whole social ethical code. Sometimes this is done without explaining what "good" is, since it is considered obvious, or perhaps more convenient to the priesthood to leave it unspecified. (Read the Koran for an extreme example.)
The key contrast is between "good" people who will do the moral thing, and "evil" people who do just the opposite.
Turning an evil person into a good person can be done by reasoning with them, teaching them willpower, or convincing them they will be punished for being evil.
Ethical judgements are different from utility judgements. Utility is a tool of reason, and reason only tells you how to get what you want, whereas ethics tells you what you ought to want. Therefore utilitarians are unethical.
Human society requires spiritual guidance and physical force to stop people from using reason to seek their own utility.
- Religion is necessary even if it is false.
- Reason must be strictly subordinated to spiritual authority.
- Smart people are less moral than dumb people, because reason maximizes personal utility.
Since ethics are desirable, and yet contrary to human reason, they prove that human values transcend logic, biology, and the material world, and derive from a spiritual plane of existence.
If there is no God, and no spiritual world, then there is no such thing as good.
- Sartre: "There can no longer be any good a priori, since there is no infinite and perfect consciousness to think it."
A person's ethicality is a single dimension, determined by the degree to which a person has willpower and subsumes their utility to social utility. Each person has a level of ethicality that is the same in all domains. You can be a good person, an evil person, or somewhere in between - but that's it. You should not expect someone who cheats at cards to be courageous in battle, unless they really enjoy battle.

People do choose whether to follow the ethics society promulgates. And they must weigh their personal satisfaction against the satisfaction of others; and those weights are probably relatively constant across domains for a given person. So there is some truth in the standard view. I want to point out errors; but I mostly want to change the focus. The standard view focuses on a person struggling to implement an ethical system, and obliterates distinctions between the ethics of that person, the ethics of society, and "true" ethics (whatever they may be). I will call these "personal ethics", "social ethics", and "normative ethics" (although the last encompasses all of the usual meaning of "ethics", including meta-ethics). I want to increase the emphasis on personal ethics, or ethical intuitions. Mostly just to insist that they exist. (A surprising number of people simultaneously claim to have strong moral feelings, and that people naturally have no moral feelings.)

The conventional story denies these first two exist: Ethics is what is good; society tries to figure out what is good; and a person is more or less ethical to the degree that they act in accordance with ethics.

The chief error of the standard view is that it explains ethics as a war between the physical and the spiritual. If a person is struggling between doing the "selfish" thing and the "right" thing, that proves that they want both about equally. The standard view instead supposes that they have a physical nature that wants only the "selfish" thing, and some internal or external spiritual force pulling them towards the "right" thing. It isn't interested in the detailed differences between ethical beliefs of different cultures, or different individuals. It may quantify them, to show they are not numerous, and thus "prove" that ethics are (and should be) universal. Or it may average them together, to give a closer estimate of the one true ethics. It thus hinders people from asking questions about the evolutionary stability of different ethical systems, or how society might use something analogous to a balance of power or a parliamentary system to produce order out of a variety of ethical systems. The quest for perfection dismisses all natural ethics as unsatisfactory. ("Natural law" is not the study of natural ethics, but the attempt to find the one true ethics in nature.) And it emphasizes some ways of influencing a persons' ethics to the exclusion of other ways.

You could recast this as the conscious mind taking the place of the spiritual nature, and the subconscious mind taking the place of the physical nature; and willpower is the exertion of control over the subconscious by the conscious. (Suggested by my misinterpretation of Matt's comment.) But to use that to defend the "ethics as willpower" view, you assume that the subconscious usually wants to do immoral things, while the conscious mind is the source of morality. And I have no evidence that my subconscious is less likely to propose moral actions than my conscious. My subconscious mind usually wants to be nice to people; and my conscious mind sometimes comes up with evil plans that my subconscious responds to with disgust.

... but being evil is harder than being good

At times, I've rationally convinced myself that I was being held back from my goals by my personal ethics, and I determined to act less ethically. Sometimes I succeeded. But more often, I did not. Even when I did, I had to first build up a complex structure of rationalizations, and exert a lot of willpower to carry through. I have never been able (or wanted) to say, "Now I will be evil" (by my personal ethics) and succeed.

If being good takes willpower, why does it take more willpower to be evil?

Ethics as innate

One theory that can explain why being evil is hard, says that people are noble savages by birth, and would enact the true ethics if only their inclinations were not crushed by society.² If you have friends who have raised their children by this theory, I probably need say no more.

A non-absolutist version of this theory would be almost the same as my theory; but my theory is informed by evolutionary theory, and human child-rearing is also part of our evolutionary history. Our genes can't be guaranteed to be adaptive if that powerful environmental influence is removed. This means that the "ethics as innate" theory has vastly different consequences for child-rearing.

Ethics as taste

Try this on instead: Think of the intuitions underlying your personal morality as the same sort of thing as your personal taste in food. It's hard to learn not to like ice cream. But it's easy to learn to like Brussels sprouts, Lima beans, beer, cigarettes, coffee, Cabernet, or other "acquired tastes". You can learn to like, or not to hate, some outcomes by acclimatizing yourself to them.

We may use the word "morality" only for interactions with other agents; but we can appreciate that it is just part of a broader underlying cognitive activity. This activity is the cognitive end of taste, where you can't just try everything out and see what you like best; you need to stop and figure out in advance what actions will usually produce the most pleasing outcomes. This often involves interactions with other people, because people are complicated. But it can also involve other complex decisions, such as whether to increase or decrease taxes, pass or punt on the fourth down, or use emacs or vim. It's not surprising that our emotions about these decisions often resemble our emotions about moral decisions.

The important advantage, to a rationalist, is that rationality and morals are no longer separate magisteria. We don't need separate models of rational behavior and moral behavior, and a way of resolving conflicts between them. If you are using utility functions, you only need one model; values of all types go in, and a single utility comes out. (If you are not using utility functions, use whatever it is you use to predict human behavior. The point is that you only need one of them.) It's true that we have separate neural systems that respond to different classes of situation; but no one has ever protested against a utility-based theory of rationality by pointing out that there are separate neural systems responding to images and sounds, and so we must have separate image-values and sound-values and some way of resolving conflicts between image-utility and sound-utility. The division of utility into moral values and all other values may even have a neural basis; but modelling that difference has, historically, caused much greater problems than it has solved.

The problem for this theory is: If ethics is just taste, why are we nice to each other? The answer comes from evolutionary theory. Exactly how it does this is controversial, but it is no longer a deep mystery. One feasible answer is that reproductive success is proportional to inclusive fitness.³ It is important to know how much of our moral intuitions is innate, and how much is conditioned; but I have no strong opinion on this other than that it is probably some of each.

This theory has different implications than the standard story:

Behaving morally feels good.
Social morals are encouraged by creating conditions that bring personal morals into line with social morals.
A person can have personal morals similar to society's in one domain, and very different in another domain.
A person learns their personal morals when they are young.
Being smarter enables you to be more ethical.
A person will come to feel that an action is ethical if it leads to something pleasant shortly after doing it, and unethical if it leads to displeasure.
A person can extinguish a moral intuition by violating it many times without consequences - whether they do this of their own free will, or under duress.
It may be easier to learn to enjoy new ethical behaviors (thou shalts), than to dislike enjoyable behaviors (thou shalt nots).
The key contrast is between "good" people who want to do the moral thing, and "bad" people who are apathetic about it.
Turning a (socially) bad person into a good person is done one behavior at a time.
Society can reason about what ethics they would like to encourage under current conditions.

As I said, this is nothing new. The standard story makes concessions to it, as social conservatives believe morals should be taught to children using behaviorist principles ("Spare the rod and spoil the child"). This is the theory of ethics endorsed by "Walden Two" and warned against by "A Clockwork Orange". And it is the theory of ethics so badly abused by the former Soviet Union, among other tyrannical governments. More on this, hopefully, in a later post.

Does that mean I can have all the pie?

No.

Eliezer addressed something that sounds like the "ethics as taste" theory in his post "Is morality preference?", and rejected it. However, the position he rejected was the straw-man position that acting to immediately gratify your desires is moral behavior. (The position he ultimately promoted, in "The meaning of right", seems to be the same I am promoting here: That we have ethical intuitions because we have evolved to compute actions as preferable that maximized our inclusive fitness.)

Maximizing expected utility is not done by greedily grabbing everything within reach that has utility to you. You may rationally leave your money in a 401K for 30 years, even though you don't know what you're going to do with it in 30 years and you do know that you'd really like a Maserati right now. Wanting the Maserati does not make buying the Maserati rational. Similarly, wanting all of the pie does not make taking all of the pie moral.

More importantly, I would never want all of the pie. It would make me unhappy to make other people go hungry. But what about people who really do want all of the pie? I could argue that they reason that taking all the pie would incur social penalties. But that would result in morals that vanish when no one is looking. And that's not the kind of morals normal people have.

Normal people don't calculate the penalties they will incur from taking all the pie. Sociopaths do that. Unlike the "ethics as willpower" theorists, I am not going to construct a theory of ethics that takes sociopaths as normal.⁴ They are diseased, and my theory of ethical behavior does not have to explain their behavior, any more than a theory of rationality has to explain the behavior of schizophrenics. Now that we have a theory of evolution that can explain how altruism could evolve, we don't have to come up with a theory of ethics that assumes people are not altruistic.

Why would you want to change your utility function?

Many LWers will reason like this: "I should never want to change my utility function. Therefore, I have no interest in effective means of changing my tastes or my ethics."

Reasoning this way makes the distinction between ethics as willpower and ethics as taste less interesting. In fact, it makes the study of ethics in general less interesting - there is little motivation other than to figure out what your ethics are, and to use ethics to manipulate others into optimizing your values.

You don't have to contemplate changing your utility function for this distinction to be somewhat interesting. We are usually talking about society collectively deciding how to change each others' utility functions. The standard LessWrongian view is compatible with this: You assume that ethics is a social game in which you should act deceptively, trying to foist your utility functions on other people and avoid letting yours being changed.

But I think we can contemplate changing our utility functions. The short answer is that you may choose to change your future utility function when doing so will have the counter-intuitive effect of better-fulfilling your current utility function (as some humans do in one ending of Eliezer's story about babyeating aliens). This can usually be described as a group of people all conspiring to choose utility functions that collectively solve prisoners' dilemmas, or (as in the case just cited) as a rational response to a threatened cost that your current utility function is likely to trigger. (You might model this as a pre-commitment, like one-boxing, rather than as changing your utility function. The results should be the same. Consciously trying to change your behavior via pre-commitment, however, may be more difficult, and may be interpreted by others as deception and punished.)

(There are several longer, more frequently-applicable answers; but they require a separate post.)

Fuzzies and utilons

Eliezer's post, Purchase fuzzies and utilons separately, on the surface appears to say that you should not try to optimize your utility function, but that you should instead satisfy two separate utility functions: a selfish utility function, and an altruistic utility function.

But remember what a utility function is. It's a way of adding up all your different preferences and coming up with a single number. Coming up with a single number is important, so that all possible outcomes can be ordered. That's what you need, and ordering is what numbers do. Having two utility functions is like having no utility function at all, because you don't have an ordering of preferences.

The "selfish utility function" and the "altruistic utility function" are different natural categories of human preferences. Eliezer is getting indirectly at the fact that the altruistic utility function (which gives output in "fuzzies") is indexical. That is, its values have the word "I" in them. The altruistic utility function cares whether you help an old lady across the street, or some person you hired in Portland helps an old lady across the street. If you aren't aware of this, you may say, "It is more cost-effective to hire boy scouts (who work for less than minimum wage) to help old ladies across the street and achieve my goal of old ladies having been helped across the street." But your real utility function prefers that you helped them across the street; and so this doesn't work.

Conclusion

The old religious view of ethics as supernatural and contrary to human nature is dysfunctional and based on false assumptions. Many religious people claim that evolutionary theory leads to the destruction of ethics, by teaching us that we are "just" animals. But ironically, it is evolutionary theory that provides us with the understanding we need to build ethical societies. Now that we have this explanation, the "ethics as taste" theory deserves to be evaluated again, and see if it isn't more sensible and more productive than the "ethics as willpower" theory.

0. I use the phrase "effectively believe" to mean both having a belief, and having habits of thought that cause you to also believe the logical consequences of that belief.

1. We have disagreements, such as the possibility of dividing values into terminal and instrumental, the relation of the values of the mind to the values of its organism, and whether having a value implies that propagating that value is also a value of yours (I say no). But they don't come into play here.

2. Contrary to popular opinion, this was not Rousseau's theory of human nature. (It may be a bastard popularization of Rousseau's theory that eventually killed and supplanted its parent.)

3. For more details, see Eliezer's meta-ethics sequence.

4. Also, I do not take Gandhi as morally normal. Not all brains develop as their genes planned; and we should expect as many humans to be pathologically good as are pathologically evil. (A biographical comparison between Gandhi and Hitler shows a remarkable number of similarities.)

144 comments

How to annoy misanthropes and bleeding-hearts

Thu, 07 Jul 2011 12:27:38 +1000

Submitted by PhilGoetz • 26 votes • 31 comments

Here is a simple method for resolving some arguments about free will. Not for resolving the question, mind you. Just the arguments.

One group of people doesn't want to give people any credit for anything they do. All good deeds are ultimately done for "selfish" reasons, where even having a goal of helping other people counts as selfish. The quote from Lukeprog's recent article is a perfect example:

No one deserves thanks from another about something he has done for him or goodness he has done. He is either willing to get a reward from God, therefore he wanted to serve himself. Or he wanted to get a reward from people, therefore he has done that to get profit for himself. Or to be mentioned and praised by people, therefore, it is also for himself. Or due to his mercy and tenderheartedness, so he has simply done that goodness to pacify these feelings and treat himself.

- Mohammed Ibn Al-Jahm Al-Barmaki

Another group of people doesn't want to blame people for anything they do. Criminals sometimes had criminal parents - crime was in their environment and in their genes. Or, to take a different variety of this attitude, cultural beliefs that seem horrible to us are always justifiable within their own cultural context.

The funny thing is that these are different groups. Both assert that people should not be given credit, or else blame, for their actions, beyond the degree of free will that they had. Yet you rarely find the same person who will not give people credit for their good deeds unwilling to blame them for their bad deeds, or vice-versa.

When you find yourself in an argument that appears to be about free will, but is really about credit or blame, ask the person to agree that the matter applies equally to good deeds and bad deeds - however they define those terms. This may make them lose interest in the argument - because it no longer does what they want it to do.

31 comments

Pancritical Rationalism Can Apply to Preferences and Behavior

Wed, 25 May 2011 22:06:11 +1000

Submitted by TimFreeman • 2 votes • 29 comments

ETA: As stated below, criticizing beliefs is trivial in principle, either they were arrived at with an approximation to Bayes' rule starting with a reasonable prior and then updated with actual observations, or they weren't. Subsequent conversation made it clear that criticizing behavior is also trivial in principle, since someone is either taking the action that they believe will best suit their preferences, or not. Finally, criticizing preferences became trivial too -- the relevant question is "Does/will agent X behave as though they have preferences Y", and that's a belief, so go back to Bayes' rule and a reasonable prior. So the entire issue that this post was meant to solve has evaporated, in my opinion. Here's the original article, in case anyone is still interested:

Pancritical rationalism is a fundamental value in Extropianism that has only been mentioned in passing on LessWrong. I think it deserves more attention here. It's an approach to epistemology, that is, the question of "How do we know what we know?", that avoids the contradictions inherent in some of the alternative approaches.

The fundamental source document for it is William Bartley's Retreat to Commitment. He describes three approaches to epistemology, along with the dissatisfying aspects of the other two:

Nihilism. Nothing matters, so it doesn't matter what you believe. This path is self-consistent, but it gives no guidance.
Justificationlism. Your belief is justified because it is a consequence of other beliefs. This path is self-contradictory. Eventually you'll go in circles trying to justify the other beliefs, or you'll find beliefs you can't jutify. Justificationalism itself cannot be justified.
Pancritical rationalism. You have taken the available criticisms for the belief into account and still feel comfortable with the belief. This path gives guidance about what to believe, although it does not uniquely determine one's beliefs. Pancritical rationalism can be criticized, so it is self-consistent in that sense.

Read on for a discussion about emotional consequences and extending this to include preferences and behaviors as well as beliefs.

"Criticism" here basically means philosophical discussion. Keep in mind that "criticism" as a hostile verbal interaction is a typical cause of failed relationships. If you do nothing but criticize a person, the other person will eventually find it emotionally impossible to spend much time with you. If you want to keep your relationships and do pancritical rationalism, be sure that the criticism that's part of pancriticial rationalism is understood to be offered in a helpful way, not a hostile way, and that you're doing it with a consenting adult. In particular, it has to be clear to all participants that there every available option will, in practice, have at least one valid criticism, so the goal is to choose something with criticisms you can accept, not to find something perfect.

We'll start by listing some typical criticisms of beliefs, and then move on to criticizing preferences and behaviors.

Criticizing beliefs is a special case in several ways. First, you can't judge the criticisms as true or false, since you haven't decided what to believe yet. Second, the process of criticizing beliefs is almost trivial in principle: apply Bayes' rule, starting with some reasonable prior. Neither of these special cases apply to criticizing preferences or behaviors, so pancriticial rationalism provides an especially useful framework for discussing them.

Criticizing beliefs is not trivial in practice, since there are nonrational criticisms of belief, there is more than one reasonable prior, Bayes' rule can be computationally intractable, and in practice people have preexisting non-Bayesian belief strategies that they follow.

With that said, a number of possible criticisms of a belief come to mind:

Perhaps it contains self-contradictions.
Perhaps it cannot arrived at by starting with a reasonably unbiased prior and doing updates according to Bayes' rule. (As a special case, perhaps it is contradicted by available evidence.)
Perhaps it is so structured that it is invulnerable to being changed after it is adopted, regardless of the evidence observed.
Perhaps it does not make predictions about the world.
Perhaps it is really a preference or a behavior. ("I believe in free speech" or "I believe I'll have another drink.")
Perhaps it is unpopular.
Perhaps it is inconsistent with some ancient religious book or another.

The last two of these illustrate that the weight one gives to a criticism is subjectively determined. Those last two criticisms are true for many beliefs discussed here, and the last one is true for essentially every belief if you pick the right religious book.

Once you accept the idea that beliefs can be criticized, it's a small step from there to adopting a similar approach to preferences and behavior. Here are some plausible criticisms of a preference:

Perhaps it is not consistent with your beliefs about cause-and-effect. That is, the preference prefers X over Y and also prefers the expected consequences of Y over the expected consequences of X.
Perhaps it cannot be used to actually decide what to do. There are several subcases here:
- Perhaps it has mathematical properties that break some decision theories, such as an unbounded utility. Concerns about actual known breakage or conjectured breakage are two different criticisms.
- Perhaps it is defined in such a way that what you prefer depends on things you cannot know.
- Perhaps it gives little guidance, that its, it considers many pairs of outcomes that you expect to actually encounter as equally preferable.
Perhaps the stated preference is ineffective or counterproductive as a social signal. There are several subcases here:
- Perhaps it is psychologically implausible. That is, perhaps it is so unlikely that a human would hold such a preference that stating the preference to others will lead the others to reasonably conclude that you're a liar or confused, rather than leading them to conclude that you have the given preference.
- Perhaps it does not help others to predict your behavior. For example, it may require complicated decisions based on debatable guesses about the remote consequences of one's actions.
- Perhaps it is not something that anybody else would want to cooperate with.
- Perhaps it is at cross-purposes with the specific people you want to signal to.
Perhaps the preference does not include preferring that you want to stay alive enough, so one would expect the preference to select itself out if there's enough time and selection pressure. ("Selection" here might mean biological evolution or some sort of technological process, take your pick based on your beliefs.)
Perhaps the preference does not include preferring that you accumulate enough power to actually do anything important.
If you believe in objective morality, perhaps the preference is inconsistent with objective morality. Someone who does believe in objective morality should fill in the details here.
Perhaps a preference is likely to have problems because it is held by only a non-controlling minority of the persons mind. This can happen in several ways:
- Perhaps a preference is likely to be self-deception because it is being claimed only because of a philosopical position, and not as a consequence of introspection or generalization from observed behavior.
- Perhaps a preference is likely to be self-decpetion because it is being claimed only because of introspection, and we expect introspection to yield socially convenient lies.
- Perhaps a claimed preference is likely to be poorly thought out because it arose nonverbally and has not been reflected upon.
Perhaps a preference is an overt deception, that is, the person claiming it knows they do not hold it. This criticism can be used by a person against themselves if they know they are lying and want clarity, or used by others against a person if the person is a poor liar.
Perhaps a preference has short-term terminal values that aren't also instrumental values.

We can also criticize behavior in at least the following ways:

Perhaps the behavior is not consistent with any reasonable guess about your preferences.
Perhaps the behavior is not consistent with your actual statements about your preferences.
Perhaps the behavior does not promote personal survival.
Perhaps the behavior is undesired by others, that is, others would prefer that you not do it.
Perhaps you did not take into account your own preferences about the outcome for others at the time you did the behavior.
Perhaps the behavior leads to active conflict with others, that is, in addition to it being against the preferences of others, it motivates them to act against you.
Perhaps the behavior will lead others to exploit you.
Perhaps you didn't take into account some of the important consequences of the behavior when you chose it.

In all cases, if you're doing or preferring or believing something that has a valid criticism, the response does not necessarily have to be "don't do/prefer/believe that". The response might be "In light of the alternatives I know about and the criticisms of all available alternatives, I accept that".

Of course, another response might be "I don't have time to consider any of that right now", but in that case you are at a level of urgency where this article won't be directly useful to you. You'll have to get yourself straightened out when things are less urgent and make use of that preparation when things are urgent.

Assuming this post doesn't quickly get negative karma, a reasonable next step would be to put a list of criticisms of beliefs, preferences, and behaviors on a not-yet-created LessWrong pancritical rationalism Wiki page. Posting them in comments might also be worthwhile. If someone else could take the initiative to update the Wiki, it would be great. Otherwise I would like to get to it eventually, but that probably won't happen soon.

Question for the readers: Is criticising a decision theory a useful separate category from the three listed above (beliefs, preferences, and behaviors)? If so, what criticisms are relevant?

29 comments

The Aliens have Landed!

Fri, 20 May 2011 03:09:16 +1000

Submitted by TimFreeman • 32 votes • 154 comments

"General Thud! General Thud! Wake up! The aliens have landed. We must surrender!" General Thud's assistant Fred turned on the lights and opened the curtains to help Thud wake up and confront the situation. Thud was groggy because he had stayed up late supervising an ultimately successful mission carried out by remotely piloted vehicles in some small country on the other side of the world. Thud mumbled, "Aliens? How many? Where are they? What are they doing?" General Thud looked out the window, expecting to see giant tripods walking around and destroying buildings with death rays. He saw his lawn, a bright blue sky, and hummingbirds hovering near his bird feeder.

Fred was trying to bring Thud up to speed as quickly as possible. "Thousands of them, General! 2376, to be precise. They gave us a map; we know where they all are. They aren't doing anything overt, but the problem is their computation! I have one here, if you'd like to look." Fred removed a black sphere two inches in diameter from his pocket and gave it to Thud.

Thud sat on his bed holding the small sphere and staring at it dumbfounded. "Okay, you think we should surrender to a few thousand small spheres. Why is that, exactly?" The sphere seemed a little flexible in Thud's hand. As he experimented a few seconds to see just how flexible, it collapsed in his hand, converting itself into a loose clump of alien sand that landed in his lap and started to dribble onto his bed and the floor. Thud stood up and brushed the rest of the sand off of his pyjamas and bed, and thought for a moment about where he left his vacuum cleaner bags. He was not impressed with these aliens.

Fred said "I don't think you wanted to do that, sir. Their ultimatum states that for every alien we destroy, they'll manufacture two in the outer reaches of the Solar System where we'll never find them!"

Thud said, "Okay, so now you think we should surrender to 2375 small spheres, and two or more small spheres that are out of the battlefield for the moment. Why is that?"

Fred said "Well, you remember a few years back when some people copied their brain state into a computer and posted it to the Internet? Apparently somebody copied the data across an unencrypted wireless link, the aliens picked it up with their radio telescopes, and now they are simulating those poor people in these black spheres and torturing the simulations! They sent us videos!" Fred held up his cell phone, pushed a button, and showed the video to Thud.

Thud looked at the video for a moment and said, "Yep, that's torture. Do these people know anything potentially useful to the aliens?"

Fred said, "Well, they know how to break into a laboratory that has brain scanning tools and push some buttons. That was apparently the high point of their lives. But none of that matters, the aliens don't seem to be torturing them for information anyway."

Thud was still suffering from morning brain fog. He rubbed his eyes. "And why should we surrender?"

Fred said, "The aliens have made a trillion copies of these poor people and will run the torture simulations on the little black spheres until we march all of our citizens into the death camps they demand we build! We have analyzed these black spheres and the engineering diagrams the aliens gave us, and we know this to be true. We only have ten billion citizens, and this simulated torture is much worse than simulated death, so the total utility is much greater if we surrender!"

Thud yawned. "Fred, you're fired. Get out of my house." As Fred left, Thud closed his curtains and tried to get back to sleep.

-----

Michael said "So I take it you no longer assist Thud. What are you doing now?"

Fred reclined comfortably on the analyst's couch. "I help out at the cafeteria as a short order cook. But I'm not worried about my career right now. I have nightmares about all these simulated people being tortured in the flimsy alien spheres."

"Thud surely knows the simulations are being tortured too. Do you think he has nightmares about this?"

"No, he doesn't seem to care."

"Have you always cared about the well-being of simulations?"

"No, when I was a teenager I was self-centered and conceited and didn't care about anybody else, including simulated people."

"So at some point you self-modified to care about simulations. If it helps you, you could self-modify again."

"But I don't want to!"

"Did you want to self-modify to care about simulations in the first place?"

"No, it just sort of happened as I grew up."

"Is there any logical inconsistency in Thud's position?"

Fred thought for a bit. "Not that I can see. The value one assigns to simulations seems to be an arbitrary choice. Ignoring the alien invasion certainly hasn't harmed his career."

"Concern about simulations seems to give the aliens more influence over you than Thud would prefer. What would you prefer?"

"Well, I'd also prefer the aliens not to be able to jerk me around. I really don't have room in my life for it now. In the grand scheme of things, it seems just wrong -- they shouldn't be able to genocide a species with a few thousand stupid spheres that just sit there converting sunlight to heat."

Michael passed Fred a piece of paper with a short list of bulleted items. "This is the procedure I teach my clients who want to change their preferences. After you've learned it, you can decide whether and how you want to use it..."

154 comments

Values vs. parameters

Tue, 17 May 2011 15:53:59 +1000

Submitted by PhilGoetz • 9 votes • 65 comments

I've written before about the difficulty of distinguishing values from errors, from algorithms, and from context. Now I have to add to that list: How can we distinguish our utility function from the parameters we use to apply it?

In my recent discussion post, "Rationalists don't care about the future", I showed that exponential time-discounting, plus some assumptions about physics and knowledge, leads to not caring about the future. Many people responded by saying that, if I care about the future, this shows that my utility function does not use exponential time-discounting.

This response assumes that the shape of my time-discounting function is part of my utility function. In other words, the way you time-discount is one of your values.

By contrast, Eliezer wrote an earlier post saying that we should use human values, but without time-discounting. Eliezer is aware that humans appear to use time discounting. Therefore, this implicitly claims that the time-discounting function is not one of our values. It's a parameter for how we implement them.

(Some of the arguments Eliezer used were value-based arguments, suggesting that we can use our values to set the parameters that we use to implement our values... I suspect this recursive approach could introduce bogus solutions, like multiplying both sides of an equation by a variable, or worse; but that would take a longer post to address. I will note that some recursive equations do have unique solutions.)

The program of CEV assumes that a transhuman can use some extrapolated version of values currently used by some humans. If that transhuman has a life expectancy of a billion years, it will likely view time discounting differently. Eliezer's post against time discounting suggests, to me, a God-like view of the universe, in which we eliminate time discounting in the same way (and for the same reasons) that many people want to eliminate space-discounting (not caring about far-away people) in contemporary ethics. This is taking an ethical code that evolved agents have, which is constructed to promote the propagation of those agents' genes, and applying it without reference to any particular set of genes. This is also pretty much what folk-morality says a social moral code is. So the idea that you can apply the same utility function from a radically different context, is inherent in CEV, and is common to much public discourse on ethics which assumes that you can construct a social morality that is based on the morality we find in individual agents.

On the other hand, I have argued that assuming that social ethics and individual ethics are the same, is either merely sloppy thinking, or an evolved (or deliberately constructed) lie. People who believed this would probably subscribe to a social-contract theory of ethics. (This view also has problems, beyond the scope of this post.)

I have one heuristic that I think is pretty good for telling when something is not a value: If it's mathematically wrong, it's an error, not a value. So my inclination is to point out that exponential time-discounting is correct. All other forms of time-discounting lead to inconsistencies. You can time-discount exponentially; or you can not time-discount at all, as Eliezer suggested; or you can be in error.

But my purpose in this post is not to continue the arguments from that other post. It's to point out this additional challenge in isolating what values are. Is your time-discounting function a value, or a value parameter?

65 comments

Separate morality from free will

Sun, 10 Apr 2011 12:35:22 +1000

Submitted by PhilGoetz • 6 votes • 83 comments

[I made significant edits when moving this to the main page - so if you read it in Discussion, it's different now. It's clearer about the distinction between two different meanings of "free", and why linking one meaning of "free" with morality implies a focus on an otherworldly soul.]

It was funny to me that many people thought Crime and Punishment was advocating outcome-based justice. If you read the post carefully, nothing in it advocates outcome-based justice. I only wanted to show how people think, so I could write this post.

Talking about morality causes much confusion, because most philosophers - and most people - do not have a distinct concept of morality. At best, they have just one word that composes two different concepts. At worst, their "morality" doesn't contain any new primitive concepts at all; it's just a macro: a shorthand for a combination of other ideas.

I think - and have, for as long as I can remember - that morality is about doing the right thing. But this is not what most people think morality is about!

Free will and morality

Kant argued that the existence of morality implies the existence of free will. Roughly: If you don't have free will, you can't be moral, because you can't be responsible for your actions.¹

The Stanford Encyclopedia of Philosophy says: "Most philosophers suppose that the concept of free will is very closely connected to the concept of moral responsibility. Acting with free will, on such views, is just to satisfy the metaphysical requirement on being responsible for one's action." ("Free will" in this context refers to a mysterious philosophical phenomenological concept related to consciousness - not to whether someone pointed a gun at the agent's head.)

I was thrown for a loop when I first came across people saying that morality has something to do with free will. If morality is about doing the right thing, then free will has nothing to do with it. Yet we find Kant, and others, going on about how choices can be moral only if they are free.

The pervasive attitudes I described in Crime and Punishment threw me for the exact same loop. Committing a crime is, generally, regarded as immoral. (I am not claiming that it is immoral. I'm talking descriptively about general beliefs.) Yet people see the practical question of whether the criminal is likely to commit the same crime again, as being in conflict with the "moral" question of whether the criminal had free will. If you have no free will, they say, you can do the wrong thing, and be moral; or you can do the right thing, and not be moral.

The only way this can make sense, is if morality does not mean doing the right thing. I need the term "morality" to mean a set of values, so that I can talk to people about values without confusing both of us. But Kant and company say that, without free will, implementing a set of values is not moral behavior. For them, the question of what is moral is not merely the question of what values to choose (although that may be part of it). So what is this morality thing?

Don't judge my body - judge my soul

My theory #1: Most people think that being moral means acting in a way that will earn you credit with God.

When theory #1 holds, "being moral" is shorthand for "acting in your own long-term self-interest". Which is pretty much the opposite of what we usually pretend being moral means.

(Finding a person who believes free will is needed for morality, and also that one should be moral even if neither God nor the community could observe, does not disprove that theory #1 is a valid characterization of the logic behind linking morals and free will. The world is full of illogical people. My impression, however, is that the people who insist that free will is needed for morality are the same people who insist that religion is needed for morality. This makes sense, if religion is needed to provide an observer to provide credit.)

My less-catchy but more-general theory #2, which includes #1 as a special case: Most people conceive of morality in a way that assumes soul-body duality. This also includes people who don't believe in a God who rewards and punishes in the afterlife, but still believe in a soul that can be virtuous or unvirtuous independent of how virtuous the body it is encased in is.

When you see (philosophical) free will being made a precondition for moral behavior, it means that the speaker is not concerned with doing the right thing. They are concerned with winning transcendent virtue points for their soul.

Moral behavior is intentional, but need not be free

I think both sides agree that morality has to do with intentions. You can't be moral unintentionally. That's because morality is (again, AFAIK we all agree) a property of a cognitive agent, not a property of the agent and its environment. Something that an agent doesn't know about its environment has no impact on whether we judge that agent's actions to be moral. Knowing the agent's intentions helps us know if this is an agent that we can expect to do the right thing in the future. But computers, machines, even thermostats, can have intentions ascribed to them. To decide how we should be disposed towards these agents, we don't need to worry about the phenomenological status of these intentions, or whether there are quantum doohickeys in their innards giving them free will. Just about what they're likely to do.

If people were concerned with doing the right thing, and getting credit for it in this world, they would only need to ask about an agent's intentions. They would care whether Jim's actions were free in the "no one pointed a gun at him and made him do it" sense, because if Joe made Jim do it, then Joe should be given the credit or blame. But they wouldn't need to ask whether Jim's intentions were free in the "free will vs. determinism" or "free will vs. brain deficiency" sense. Having an inoperable brain condition would not affect how we used a person's actions to predict whether they were likely to do similar things in the future - they're still going to have the brain condition. We only change our credit assignment due to a brain condition if we are trying to assign credit to the non-physical part of a person (their soul).

(At this point I should also mention theory #3: Most people fail to distinguish between "done with philosophical free will" and "intentional". They thus worry about philosophical free will when they mean to worry about intention.)²

Why we should separate the concepts of "morality" and "free will"

The majority opinion of what a word means is, by definition, the descriptively correct usage of the word. I'm not arguing that the majority usage is descriptively wrong. I'm arguing that it's prescriptively wrong, for these reasons:

It isn't parsimonious. It confuses the question of figuring out what values are good, and what behaviors are good, with the philosophical problem of free will. Each of these problems is difficult enough on its own!
It is inconsistent with our other definitions. People map questions about what is right and wrong onto questions about morality. They will get garbage out of their thinking if that concept, internally, is about something different. They end up believing there are no objective morals - not necessarily because they've thought it through logically, but because their conflicting definitions make them incapable of coherent thought on the subject.
It implies that morality is impossible without free will. Since a lot of people on LW don't believe in free will, they would conclude that they don't believe in morality if they subscribed to Kant's view.
When questions of blame and credit take center stage, people lose the capacity to think about values. This is demonstrated by some Christians who talk a lot about morality, but assume, without even noticing they're doing it, that "moral" is a macro for "God said do this". They failed to notice that they had encoded two concepts into one word, and never got past the first concept.

For morality to be about oughtness, so that we are able to reason about values, we need to divorce it completely from free will. Free will is still an interesting and possibly important problem. But we shouldn't mix it in together with the already-difficult-enough problem of what actions and values are moral.

1. I am making the most-favorable re-interpretation. Kant's argument is worse, as it takes a nonsensical detour from morality, through rationality, back to free will.

2. This is the preferred theory under, um, Goetz's Cognitive Razor: Prefer the explanation for someone's behavior that supposes the least internal complexity of them.

83 comments

Crime and punishment

Fri, 25 Mar 2011 08:53:09 +1100

Submitted by PhilGoetz • 38 votes • 189 comments

Why do those words go together?

Society - and for once, I'm using this term universally - teaches that, if you committed a crime, you should be punished.

But in some societies, we have an insanity defense. If you had a brain condition so that you had no - here it's a little vague - consciousness, or moral sense, or free will, or, well, something - then it would be cruel to punish you for your crime. Instead of going to prison, you should be placed somewhere where you can't hurt anybody, where professional physicians and counselors can study your case and try to reform you so that you can rejoin society.

Wait - so that isn't what prison is for?

No. Prison is to punish people. Is it any wonder that prison doesn't reform people, when we don't want it to reform them? Most people would be upset if prisoners could go in on Friday, and emerge, rehabilitated, on Sunday. When people say, "It would be cruel to punish people who aren't responsible for their actions", they are implicitly saying, "Prison is necessarily cruel; and that's good, because we should be cruel to criminals who are responsible for their actions."

But the more we learn about psychology and neuroscience, the further responsibility recedes into the distance.

Outcome-based justice argues that we should give up playing the blame game, because neuroscience keeps finding more and more proofs that things are "not our fault". Instead, we should write laws that deter crime.

You might think this is what we already try to do. But it isn't! Witness this confused article from the Brookings Institute, Cognitive Neuroscience and the Future of Punishment by O. Carter Snead. Snead objects to outcome-based justice. He summarized all of the arguments for it, yet managed to completely miss their point, concluding where he started from, saying that outcome-based justice is obviously bad because it could lead to being cruel to people who didn't deserve it. (Instead of only being cruel to the people who do deserve it, which is obviously what we want to do.)

Snead understands that outcome-based justice deters crime:

Many features of the criminal justice system that are frequently criticized as draconian and inhumane are, in fact, motivated by a purely consequentialist crime-control rationale. Such measures include laws that authorize life sentences for recidivists (e.g., “three strikes” laws), laws that reduce the age at which offenders can be tried as adults, laws that punish gang membership, laws that require the registration of sex offenders, laws that dramatically increase sentences by virtue of past history, and, most paradigmatically, laws that provide for the involuntary civil commitment of sexual offenders who show difficulty controlling their behavior.

You might expect that Snead goes on to explain why these laws are bad things. But he doesn't! He assumes we can all see that these are obviously bad things.

The Wrath of Kahneman describes a study which asked whether people punish others in order to deter crime, and concluded, No. People are doing something else.

One theory is that people are trying to be fair. Everyone should get the same chances; everyone should get the same punishment for the same crime. John Rawls argues this explicitly in Justice as Fairness: Justice should not be utilitarian, but should instead be fair.

I believe Rawls' view is also the popular view of what "justice" means. And, I will argue in a later post, it is part of a pattern showing a deep divide between two different ways of using the word "ethics".

ADDED: Constant made the point that, while one part of outcome-based justice is preventing future harm from the criminal on the dock, another part is deterring harm by other criminals. This latter part does not benefit from punishing criminals who cannot be deterred. Thus, to optimally punish both criminals who can and cannot be deterred, the law requires a concept of moral culpability, and should punish criminals who can't be deterred more lightly. This is a better origin story for the linking together of morality and free will than the just-so story I'd come up with, so I plan on stealing it for my next post. (SilasBarta may have been trying to make the same point, but I found his comments impenetrable.)

(This post is laying groundwork for two other posts that will go in different directions, neither of which concerns justice.)

189 comments