ETA: As stated below, criticizing beliefs is trivial in principle, either they were arrived at with an approximation to Bayes' rule starting with a reasonable prior and then updated with actual observations, or they weren't.  Subsequent conversation made it clear that criticizing behavior is also trivial in principle, since someone is either taking the action that they believe will best suit their preferences, or not.  Finally, criticizing preferences became trivial too -- the relevant question is "Does/will agent X behave as though they have preferences Y", and that's a belief, so go back to Bayes' rule and a reasonable prior. So the entire issue that this post was meant to solve has evaporated, in my opinion. Here's the original article, in case anyone is still interested:

Pancritical rationalism is a fundamental value in Extropianism that has only been mentioned in passing on LessWrong. I think it deserves more attention here. It's an approach to epistemology, that is, the question of "How do we know what we know?", that avoids the contradictions inherent in some of the alternative approaches.

The fundamental source document for it is William Bartley's Retreat to Commitment. He describes three approaches to epistemology, along with the dissatisfying aspects of the other two:

• Nihilism. Nothing matters, so it doesn't matter what you believe. This path is self-consistent, but it gives no guidance.
• Justificationlism. Your belief is justified because it is a consequence of other beliefs. This path is self-contradictory. Eventually you'll go in circles trying to justify the other beliefs, or you'll find beliefs you can't jutify. Justificationalism itself cannot be justified.
• Pancritical rationalism. You have taken the available criticisms for the belief into account and still feel comfortable with the belief. This path gives guidance about what to believe, although it does not uniquely determine one's beliefs. Pancritical rationalism can be criticized, so it is self-consistent in that sense.

Read on for a discussion about emotional consequences and extending this to include preferences and behaviors as well as beliefs.

Related: Cached Selves, The Neuroscience of Desire

You don't know your own mind.
    - Jonathan Swift, Polite Conversation

Researchers showed subjects two female faces for a few seconds and asked which face was more attractive. Researchers then placed the photos face down and handed subjects the face they had chosen, asking them to explain the motives behind their choice. But sometimes, researchers used a sleight-of-hand trick to switch the photos, showing viewers the face they had not chosen. Very few subjects noticed the face they were given was not the one they had chosen. Moreover, they happily explained why they preferred the face they had actually rejected, inventing reasons like "I like her smile" even though they had actually chosen the solemn-faced picture.¹

The idea that we lack good introspective access to our own desires - that we often have no idea what we want² - is a key lemma in naturalistic metaethics, so it seems worth a post to collect the science by which we know that.

Early warnings came from split-brain research, which identified an 'interpreter' in the left hemisphere that invents reasons for beliefs and actions. When the command 'walk' was flashed to split-brain subjects' right hemispheres, they got up from their chairs and start walking away. When asked why they suddenly started walking away, they replied (for example) that they got up because they wanted a Coke.³

On January 11, 2007, I timidly whispered to myself: "There is no God."

And with that, all my Christian dreams and hopes and purposes and moral systems came crashing down.

I wrote a defiant email to the host of an atheist radio show I'd been listening to:

I was coming from a lifetime high of surrendering… my life to Jesus, releasing myself from all cares and worries, and filling myself and others with love. Then I began an investigation of the historical Jesus… and since then I’ve been absolutely miserable. I do not think I am strong enough to be an atheist. Or brave enough. I have a broken leg, and my life is much better with a crutch… I’m going to seek genuine experience with God, to commune with God, and to reinforce my faith. I am going to avoid solid atheist arguments, because they are too compelling and cause for despair. I do not WANT to live in an empty, cold, ultimately purposeless universe in which I am worthless and inherently alone.

I was not okay with the truth. I had been taught that meaning and morality and hope depended on God. If God didn't exist, then life was meaningless.

My tongue felt like cardboard for a week.

But when I pulled my head out of the sand, I noticed that millions of people were living lives of incredible meaning and morality and hope without gods. The only thing I had 'lost' was a lie, anyway.

This crisis taught me a lesson: that I could be okay with the truth.

When I realized that I am not an Unmoved Mover of my own actions, I was not much disturbed. I realized that 'moral responsibility' still mattered, because people still had reasons to condemn, praise, punish, and reward certain actions in others. And I realized that I could still deliberate about which actions were likely to achieve my goals, and that this deliberation would affect my actions. Apples didn't stop falling from trees when Einstein's equations replaced Newton's, and humans didn't stop making conscious choices that have consequences when we discovered that we are fully part of nature.

I didn't freak out when I gave up moral absolutism, either. I had learned to be okay with the truth. Whatever is meant by 'morality', it remains the case that agents have reasons to praise and condemn certain desires and actions in other agents, and that there are more reasons to praise and condemn some actions than others.

I've gone through massive reversals in my metaethics twice now, and guess what? At no time did I spontaneously acquire the urge to rape people. At no time did I stop caring about the impoverished. At no time did I want to steal from the elderly. At no time did people stop having reasons to praise or condemn certain desires and actions of mine, and at no time did I stop having reasons to praise or condemn the desires and actions of others.

We humans have a tendency to 'freak out' when our model of the world changes drastically. But we get over it.

The love a mother has for her child does not disappear when we explain the brain processes that instantiate that love. Explaining something is not explaining it away. Showing that love and happiness and moral properties are made of atoms does not mean they are just atoms. They are also love and happiness and moral properties. Water was still water after we discovered which particular atoms it was made of.

When you understand this, you need not feel the threat of nihilism as science marches on. Instead, you can jump with excitement as science locates everything we care about in the natural world and tells us how it works. Along the way, you can take joy in the merely real.

Whenever you 'lose' something as a result of getting closer to the truth, you've only lost a lie. You can face reality, even the truth about morality.

People can stand what is true,
for they are already enduring it.

- Eugene Gendlin

Part of the sequence: No-Nonsense Metaethics

When I say I think I can solve (some of) metaethics, what exactly is it that I think I can solve?

First, we must distinguish the study of ethics or morality from the anthropology of moral belief and practice. The first one asks: "What is right?" The second one asks: "What do people think is right?" Of course, one can inform the other, but it's important not to confuse the two. One can correctly say that different cultures have different 'morals' in that they have different moral beliefs and practices, but this may not answer the question of whether or not they are behaving in morally right ways.

My focus is metaethics, so I'll discuss the anthropology of moral belief and practice only when it is relevant for making points about metaethics.

So what is metaethics? Many people break the field of ethics into three sub-fields: applied ethics, normative ethics, and metaethics.

Applied ethics: Is abortion morally right? How should we treat animals? What political and economic systems are most moral? What are the moral responsibilities of businesses? How should doctors respond to complex and uncertain situations? When is lying acceptable? What kinds of sex are right or wrong? Is euthanasia acceptable?

Normative ethics: What moral principles should we use in order to decide how to treat animals, when lying is acceptable, and so on? Is morality decided by what produces the greatest good for the greatest number? Is it decided by a list of unbreakable rules? Is it decided by a list of character virtues? Is it decided by a hypothetical social contract drafted under ideal circumstances?

Metaethics: What does moral language mean? Do moral facts exist? If so, what are they like, and are they reducible to natural facts? How can we know whether moral judgments are true or false? Is there a connection between making a moral judgment and being motivated to abide by it? Are moral judgments objective or subjective, relative or absolute? Does it make sense to talk about moral progress?

Part of the sequence: No-Nonsense Metaethics

A few months ago, I predicted that we could solve metaethics in 15 years. To most people, that was outrageously optimistic. But I've updated since then. I think much of metaethics can be solved now (depending on where you draw the boundary around the term 'metaethics'.) My upcoming sequence 'No-Nonsense Metaethics' will solve the part that can be solved, and make headway on the parts of metaethics that aren't yet solved. Solving the easier problems of metaethics will give us a clear and stable platform from which to solve the hard questions of morality.

Metaethics has been my target for a while now, but first I had to explain the neuroscience of pleasure and desire, and how to use intuitions for philosophy.

Luckily, Eliezer laid most of the groundwork when he explained couldness, terminal and instrumental values, the complexity of human desire and happiness, how to dissolve philosophical problems, how to taboo words and replace them with their substance, how to avoid definitional disputes, how to carve reality at its joints with our words, how an algorithm feels from the inside, the mind projection fallacy, how probability is in the mind, reductionism, determinism, free will, evolutionary psychology, how to grasp slippery things, and what you would do without morality.

Of course, Eliezer wrote his own metaethics sequence. Eliezer and I seem to have similar views on morality, but I'll be approaching the subject from a different angle, I'll be phrasing my solution differently, and I'll be covering a different spread of topics.

Why do I think much of metaethics can be solved now? We have enormous resources not available just a few years ago. The neuroscience of pleasure and desire didn't exist two decades ago. (Well, we thought dopamine was 'the pleasure chemical', but we were wrong.) Detailed models of reductionistic meta-ethics weren't developed until the 1980s and 90s (by Peter Railton and Frank Jackson). Reductionism has been around for a while, but there are few philosophers who relentlessly play Rationalist's Taboo. Eliezer didn't write How an Algorithm Feels from the Inside until 2008.

Our methods will be familiar ones, already used to dissolve problems ranging from free will to disease. We will play Taboo with our terms, reducing philosophical questions into scientific ones. Then we will examine the cognitive algorithms that make it feel like open questions remain.

Along the way, we will solve or dissolve the traditional problems of metaethics: moral epistemology, the role of moral intuition, the is-ought gap, matters of moral psychology, the open question argument, moral realism vs. moral anti-realism, moral cognitivism vs. non-cognitivism, and more. 

You might respond, "Sure, Luke, we can do the reduce-to-algorithm thing with free will or disease, but morality is different. Morality is fundamentally normative. You can't just dissolve moral questions with Taboo-playing and reductionism and cognitive science."

Well, we're going to examine the cognitive algorithms that generate that intuition, too.

And at the end, we will see what this all means for the problem of Friendly AI.

I must note that I didn't exactly invent the position I'll be defending. After sharing my views on metaethics with many scientifically-minded people in private conversation, many have said something like "Yeah, that's basically what I think about metaethics, I've just never thought it through in so much detail and cited so much of the relevant science [e.g. recent work in neuroeconomics and the science of intuition]."

But for convenience I do need to invent a name for my theory of metaethics. I call it pluralistic moral reductionism.

Next post: What is Metaethics?

One of the central problems of Friendly Artificial Intelligence is goal system stability. Given a goal system - whether it's a utility function, a computer program, or a couple kilograms of neural tissue - we want to determine whether it's stable, meaning, is there something that might plausibly happen to it which will radically alter its behavior in a direction we don't like? As a first step in solving this problem, let's consider a classic example of goal systems that is not stable.

Suppose you are a true Bentham-Mill Utilitarian, which means you hold that the right thing to do is that which maximizes the amount of happiness minus the amount of pain in the world, summed up moment by moment. Call this HapMax for short. You determine this by assigning each person a happiness-minus-pain score at each moment, based on a complex neurological definition, and adding up the scores of each person-moment. One day, you are interrupted from your job as an antidepressant research chemist by a commotion outside. Rushing out to investigate, you find a hundred-foot tall monster rampaging through the streets of Tokyo, which says:

"I am a Utility Monster. Robert Nozick grew me in his underwater base, and now I desire nothing more than to eat people. This makes me very happy, and because I am so very tall and the volume of my brain's reward center grows with the cube of my height, it makes me *so* happy that it will outweigh the momentary suffering and shortened lifespan of anyone I eat."

As a true HapMaxer (not to be confused with a human, who might claim to be a HapMaxer but can't actually be one), you find this very convincing: the right thing to do is to maximize the number of people the monster can eat, so you heroically stand in front of the line of tanks that is now rolling down main street to buy it time. HapMax seemed like a good idea at first, but this example shows that it is very wrong. What lessons should we learn before trying to build another utility function? HapMax starts by dividing the world into pieces, and the trouble starts when one of those agents doesn't behave as expected.

Barring a major collapse of human civilization (due to nuclear war, asteroid impact, etc.), many experts expect the intelligence explosion Singularity to occur within 50-200 years.

That fact means that many philosophical problems, about which philosophers have argued for millennia, are suddenly very urgent.

Those concerned with the fate of the galaxy must say to the philosophers: "Too slow! Stop screwing around with transcendental ethics and qualitative epistemologies! Start thinking with the precision of an AI researcher and solve these problems!"

If a near-future AI will determine the fate of the galaxy, we need to figure out what values we ought to give it. Should it ensure animal welfare? Is growing the human population a good thing?

But those are questions of applied ethics. More fundamental are the questions about which normative ethics to give the AI: How would the AI decide if animal welfare or large human populations were good? What rulebook should it use to answer novel moral questions that arise in the future?

But even more fundamental are the questions of meta-ethics. What do moral terms mean? Do moral facts exist? What justifies one normative rulebook over the other?

The answers to these meta-ethical questions will determine the answers to the questions of normative ethics, which, if we are successful in planning the intelligence explosion, will determine the fate of the galaxy.

Eliezer Yudkowsky has put forward one meta-ethical theory, which informs his plan for Friendly AI: Coherent Extrapolated Volition. But what if that meta-ethical theory is wrong? The galaxy is at stake.

Princeton philosopher Richard Chappell worries about how Eliezer's meta-ethical theory depends on rigid designation, which in this context may amount to something like a semantic "trick." Previously and independently, an Oxford philosopher expressed the same worry to me in private.

Eliezer's theory also employs something like the method of reflective equilibrium, about which there are many grave concerns from Eliezer's fellow naturalists, including Richard Brandt, Richard Hare, Robert Cummins, Stephen Stich, and others.

My point is not to beat up on Eliezer's meta-ethical views. I don't even know if they're wrong. Eliezer is wickedly smart. He is highly trained in the skills of overcoming biases and properly proportioning beliefs to the evidence. He thinks with the precision of an AI researcher. In my opinion, that gives him large advantages over most philosophers. When Eliezer states and defends a particular view, I take that as significant Bayesian evidence for reforming my beliefs.

Rather, my point is that we need lots of smart people working on these meta-ethical questions. We need to solve these problems, and quickly. The universe will not wait for the pace of traditional philosophy to catch up.

A tenet of traditional rationality is that you can't learn much about the world from armchair theorizing. Theory must be epiphenomenal to observation-- our theories are functions that tell us what experiences we should anticipate, but we generate the theories from *past* experiences. And of course we update our theories on the basis of new experiences. Our theories respond to our evidence, usually not the other way around. We do it this way because it works better then trying to make predictions on the basis of concepts or abstract reasoning. Philosophy from Plato through Descartes and to Kant is replete with failed examples of theorizing about the natural world on the basis of something other than empirical observation. Socrates thinks he has deduced that souls are immortal, Descartes thinks he has deduced that he is an immaterial mind, that he is immortal, that God exists and that he can have secure knowledge of the external world, Kant thinks he has proven by pure reason the necessity of Newton's laws of motion.

These mistakes aren't just found in philosophy curricula. There is a long list of people who thought they could deduce Euclid's theorems as analytic or a priori knowledge. Epicycles were a response to new evidence but they weren't a response that truly privileged the evidence. Geocentric astronomers changed their theory *just enough* so that it would yield the right predictions instead of letting a new theory flow from the evidence. Same goes for pre-Einsteinian theories of light. Same goes for quantum mechanics. A kludge is a sign someone is privileging the hypothesis. It's the same way many of us think the Italian police changed their hypothesis explaining the murder of Meredith Kercher once it became clear Lumumba had an alibi and Rudy Guede's DNA and hand prints were found all over the crime scene. They just replaced Lumumba with Guede and left the rest of their theory unchanged even though there was no longer reason to include Knox and Sollecito in the explanation of the murder. These theories may make it over the bar of traditional rationality but they sail right under what Bayes theorem requires.

Most people here get this already and many probably understand it better than I do. But I think it needs to be brought up in the context of our ongoing discussion of normative ethics.

Unless we have reason to think about ethics differently, our normative theories should respond to evidence in the same way we expect our theories in other domains to respond to evidence. What are the experiences that we are trying to explain with our ethical theories? Why bother with ethics at all? What is the mystery we are trying to solve? The only answer I can think of is our ethical intuitions. When faced with certain situations in real life or in fiction we get strong impulses to react in certain ways, to praise some parties and condemn others. We feel guilt and sometimes pay amends. There are some actions which we have a visceral abhorrence of.

These reactions are for ethics what measurements of time and distance are for physics -- the evidence.