I'm basing my the views I express in this post primarily off Derek Parfit’s writing, specifically his book On What Matters. For this reason, it seems pretty plausible to me that there are some important points I've missed by reading too narrowly. In addition, it also seems likely that some of the ways in which I talk about particular issues around normativity will sound a bit foreign or just generally “off” to people who are highly familiar with some of these issues. One unfortunate reason for this is that the study of normative questions and of the nature of normativity seems to me to be spread out pretty awkwardly across the field of philosophy, with philosophers in different sub-disciplines often discussing apparently interconnected questions in significant isolation of one another while using fairly different terminology. This means that (e.g.) meta-ethics and decision theory are seldom talked about at the same time and are often talked about in ways that make it difficult to see how they fit together. A major reason I am leaning on Parfit’s work is that he is -- to my knowledge -- one of relatively few philosophers to have tried to approach questions around normativity through a single unified framework. ↩︎

This is a point that is also discussed at length in David Enoch’s book Taking Morality Seriously (pgs. 70-73):

Perhaps...we are essentially deliberative creatures. Perhaps, in other words, we cannot avoid asking ourselves what to do, what to believe, how to reason, what to care about. We can, of course, stop deliberating about one thing or another, and it’s not as if all of us have to be practical philosophers (well, if you’re reading this book, you probably are, but you know what I mean). It’s opting out of the deliberative project as a whole that may not be an option for us….

[Suppose] law school turned out not to be all you thought it would be, and you no longer find the prospects of a career in law as exciting as you once did. For some reason you don’t seem to be able to shake off that old romantic dream of studying philosophy. It seems now is the time to make a decision. And so, alone, or in the company of some others you find helpful in such circumstances, you deliberate. You try to decide whether to join a law firm, apply to graduate school in philosophy, or perhaps do neither.

The decision is of some consequence, and so you resolve to put some thought into it. You ask yourself such questions as: Will I be happy practicing law? Will I be happier doing philosophy? What are my chances of becoming a good lawyer? A good philosopher? How much money does a reasonably successful lawyer make, and how much less does a reasonably successful philosopher make? Am I, so to speak, more of a philosopher or more of a lawyer? As a lawyer, will I be able to make a significant political difference? How important is the political difference I can reasonably expect to make? How important is it to try and make any political difference? Should I give any weight to my father’s expectations, and to the disappointment he will feel if I fail to become a lawyer? How strongly do I really want to do philosophy? And so on. Even with answers to most – even all – of these questions, there remains the ultimate question. “All things considered”, you ask yourself, “what makes best sense for me to do? When all is said and done, what should I do? What shall I do?”

When engaging in this deliberation, when asking yourself these questions, you assume, so it seems to me, that they have answers. These answers may be very vague, allow for some indeterminacy, and so on. But at the very least you assume that some possible answers to these questions are better than others. You try to find out what the (better) answers to these questions are, and how they interact so as to answer the arch-question, the one about what it makes most sense for you to do. You are not trying to create these answers. Of course, in an obvious sense what you will end up doing is up to you (or so, at least, both you and I are supposing here). And in another, less obvious sense, perhaps the answer to some of these questions is also up to you. Perhaps, for instance, how happy practicing law will make you is at least partly up to you. But, when trying to make up your mind, it doesn’t feel like just trying to make an arbitrary choice. This is just not what it is like to deliberate. Rather, it feels like trying to make the right choice. It feels like trying to find the best solution, or at least a good solution, or at the very least one of the better solutions, to a problem you’re presented with. What you’re trying to do, it seems to me, is to make the decision it makes most sense for you to make. Making the decision is up to you. But which decision is the one it makes most sense for you to make is not. This is something you are trying to discover, not create. Or so, at the very least, it feels like when deliberating.

Specifically, the two relevant views can be described as realism and anti-realism with regard to “normativity.” We can divide the domain of “normativity” up into the domains of “practical rationality,” which describes what actions people should take, and “epistemic rationality,” which describes which beliefs or degrees of belief people should hold. The study of ethics, decision-making under uncertainty, and so on can then all be understood as sub-components of the study of practical rationality. For example, one view on the study of ethics is that it is the study of how factors other than one’s own preferences might play roles in determining what actions one should take. It should be noted that terminology varies very widely though. For example, different authors seem to use the word "ethics" more or less inclusively. The term "moral realism" also sometimes means roughly the same thing as "normative realism," as I've defined it here, and sometimes picks out a more specific position. ↩︎

An an edit to the initial post, I think it's probably worth saying more about the concept of "moral realism" in relation to "normative realism." Depending on the context, "moral realism" might be taken to refer to: (a) normative realism, (b) realism about practical rationality (not just epistemic rationality), (c) realism about practical rationality combined with the object-level belief that people should do more than just try to satisfy their own personal preferences, or (d) something else in this direction.

One possible reason the term lacks a consensus definition is that, perhaps surprisingly, many contemporary "moral realists" aren't actually very preocuppied with the concept of "morality." Popular books like Taking Morality Seriously, On What Matters, and The Normative Web spend most of their energy defending normative realism, more broadly, and my impression is that their critics spend most of their energy attacking normative realism more broadly. One reason for this shift in focus toward normative realism is the realization that, on almost any conception of "moral realism," nearly all of the standard metaphysical and epistemological objections to "moral realism" also apply just as well to normative realism in general. Another reason is that any possible distinction between moral and normative-but-not-moral facts doesn't seem like it could have much practical relevance: If we know that we should make some decision, then we know that we should take it; we have no obvious additional need to know or care whether this normative fact warrants the label "moral fact" or not. Here, for example, is David Enoch, in Taking Morality Seriously, on the concept of morality (pg. 86):

What more...does it take for a normative truth (or falsehood) to qualify as moral? Morality is a particular instance of normativity, and so we are now in effect asking about its distinctive characteristics, the ones that serve to distinguish between the moral and the rest of the normative. I do not have a view on these special characteristics of the moral. In fact, I think that for most purposes this is not a line worth worrying about. The distinction within the normative between the moral and the non-moral seems to me to be shallow compared to the distinction between the normative and the non-normative - both philosophically, and, as I am about to argue, practically. (Once you know you have a reason to X and what this reason is, does it really matter for your deliberation whether it qualifies as a moral reason?)

There are two major strands of anti-realism. Error theory (sometimes equated with “nihilism”) asserts that all claims that people should do particular things or refrain from doing particular things are false. Non-cognitivism asserts that utterances of the form “A should do X” typically cannot even really be understood as claims; they're not the sort of thing that could be true or false. ↩︎

In this post, for simplicity, I’m talking about normativity using binary language. Either it’s the case that you “should” take an action or it’s not the case that you “should” take it. But we might also talk in less binary terms. For example, there may be some actions that you merely have “more reason” to take than others. ↩︎

In Sepielli’s account, for example, the experience of feeling extremely in favor of blaming someone a little bit for taking an action X is analogous to the experience of being extremely confident that it is a little bit wrong to take action X. This account is open to at least a few objections, such as the objection that degrees of favorability don’t -- at least at first glance -- seem to obey the standard axioms of probability theory. Even if we do accept the account, though, I still feel unclear about the proper method and justification for converting debates around normative uncertainty into debates around these other kinds of psychological states. ↩︎

If my memory is correct, one example of a context in which I have encountered this subjectivist viewpoint is in a CFAR workshop. One lesson instructs attendees that if it seems like they “should” do something, but then upon reflection they realize they don’t want to do it, then it’s not actually true that they should do it. ↩︎

The PhilPapers survey suggests that about a quarter of both normative ethicists and applied ethicists also self-identify as anti-realists, with the majority of them presumably leaning toward non-cognitivism over error theory. It’s still an active matter of debate whether non-cognitivists have sensible stories about what people are trying to do when they seem to be discussing normative claims. For example, naive emotivist theories stumble in trying to explain sentences like: “It's not true that either you should do X or you should do Y.” ↩︎

There is also non-normative research that falls under the label “decision theory,” which focuses on exploring the ways in which people do in practice make decisions or neutrally exploring the implications of different assumptions about decision-making processes. ↩︎

Arguably, even in academic literature, decision theories are often discussed under the implicit assumption that some form of subjectivism is true. However, it is also very easy to modify the theories to be compatible with theories that tell you to take into account things beyond your current desires. Value might be equated with one’s future welfare, for example, or with the total future welfare of all conscious beings. ↩︎

One thing that makes this issue a bit complicated is that rationalist community writing on decision theory sometimes seems to switch back and forth between describing decision theories as normative claims about decisions (which I believe is how academic philosophers typically describe decision theories) and as algorithms to be used (which seems to be inconsistent with how academic philosophers typically describe decision theories). I think this tendency to switch back and forth between describing decision theories in these two distinct ways can be seen both in papers proposing new decision theories and in online discussions. I also think this switching tendency can make things pretty confusing. Although it makes sense to discuss how an algorithm “performs” when “implemented,” once we specify a sufficiently precise performance metric, it does not seem to me to make sense to discuss the performance of a normative claim. I think the tendency to blur the distinction between algorithms and normative claims -- or, as Will MacAskill puts it in his recent and similar critique, between "decision procedures" and "criteria of rightness" -- partly explains why proponents of FDT and other new decision theories have not been able to get much traction with academic decision theorists. For example, causal decision theorists are well aware that people who always takes the actions that CDT says they should take will tend to fare less well in Newcomb scenarios than people who always take the actions that EDT says they should take. Causal decision theorists are also well aware that that there are some scenarios -- for example, a Newcomb scenario with a perfect predictor and the option to get brain surgery to pre-commit yourself to one-boxing -- in which there is no available sequence of actions such that CDT says you should take each of the actions in the sequence. If you ask a causal decision theorist what sort of algorithm you should (according to CDT) put into an AI system that will live in a world full of Newcomb scenarios, if the AI system won’t have the opportunity to self-modify, then I think it's safe to say a causal decision theorist won’t tell you to put in an algorithm that only produces actions that CDT says it should take. This tells me that we really can’t fluidly switch back and forth between making claims about the correctness of normative principles and claims about the performance of algorithms, as though there were a canonical one-to-one mapping between these two sorts of claims. Insofar as rationalist writing on decision theory tends to do this sort of switching, I suspect that it contributes to confusion on the part of many academic readers. See also this blog post by an academic decision theorist, Wolfgang Schwarz, for a much more thorough perspective on why proponents of FDT may be having difficulty getting traction within the academic decision theory community. ↩︎

A similar concern also leads me to assign low (p<10%) probability to normative decision theory research ultimately being useful for avoiding large-scale accidental harm caused by AI systems. It seems to me like the question “What is the correct decision theory?” only has an answer if we assume that realism is true. But even if we assume that realism is true, we are now asking a normative question (“What criterion determines whether an action is one an agent ‘should’ take?”) as a way of trying to make progress on a non-normative question (“What approaches to designing advanced AI systems result in unintended disasters and which do not?”). Proponents of CDT and proponents of EDT do not actually disagree on how any given agent will behave, on what the causal outcome of assigning an agent a given algorithm will be, or on what evidence might be provided by the choice to assign an agent a given algorithm; they both agree, for example, about how much money different agents will tend to earn in the classic Newcomb scenario. What decision theorists appear to disagree about is a seperate normative question that floats above (or rather “supervenes” upon) questions about observed behavior or questions about outcomes. I don't see how answering this normative question could help us much in answering the non-normative question of what approaches to designing advanced AI systems don’t (e.g.) result in global catastrophe. Put another way, my concern is that the strategy here seems to rely on the hope that we can derive an “is” from an “ought.”

However, in keeping with the above endnote, community work on decision theory only sometimes seems to be pitched (as it is in the abstract of this paper) as an exploration of normative principles. It is also sometimes pitched as an exploration of how different “algorithms” “perform” across relevant scenarios. This exploration doesn't seem to me to have any direct link to the core academic decision theory literature and, given a sufficiently specific performance metric, does not seem to be inherently normative. I'm actually more optimistic, then, about this line of research having implications for AI development. Nonetheless, for reasons similar to the ones described in the post “Decision Theory Anti-Realism,” I'm still not very optimistic. In the cases that are being considered, the answer to the question “Which algorithm performs best?” will depend on subtle variations in the set of counterfactuals we consider when judging performance; different algorithms come out on top for different sets of counterfactuals. For example, in a prisoner’s dilemma, the best-performing algorithm will vary depending on whether we are imaging a counterfactual world where just one agent was born with a different algorithm or a counterfactual world where both agents were born with different algorithms. It seems unclear to me where we go from here except perhaps to list several different sets of imaginary counterfactuals and note which algorithms perform best relative to them.

Wolfgang Schwarz and Will MacAskill also make similar points, regarding the sensitivity of comparisons of algorithmic performance, in their essays on FDT. Schwarz writes:

Yudkowsky and Soares constantly talk about how FDT "outperforms" CDT, how FDT agents "achieve more utility", how they "win", etc. As we saw above, it is not at all obvious that this is true. It depends, in part, on how performance is measured. At one place, Yudkowsky and Soares are more specific. Here they say that "in all dilemmas where the agent's beliefs are accurate [??] and the outcome depends only on the agent's actual and counterfactual behavior in the dilemma at hand -- reasonable constraints on what we should consider "fair" dilemmas -- FDT performs at least as well as CDT and EDT (and often better)". OK. But how we should we understand "depends on ... the dilemma at hand"? First, are we talking about subjunctive or evidential dependence? If we're talking about evidential dependence, EDT will often outperform FDT. And EDTers will say that's the right standard. CDTers will agree with FDTers that subjunctive dependence is relevant, but they'll insist that the standard Newcomb Problem isn't "fair" because here the outcome (of both one-boxing and two-boxing) depends not only on the agent's behavior in the present dilemma, but also on what's in the opaque box, which is entirely outside her control. Similarly for all the other cases where FDT supposedly outperforms CDT. Now, I can vaguely see a reading of "depends on ... the dilemma at hand" on which FDT agents really do achieve higher long-run utility than CDT/EDT agents in many "fair" problems (although not in all). But this is a very special and peculiar reading, tailored to FDT. We don't have any independent, non-question-begging criterion by which FDT always "outperforms" EDT and CDT across "fair" decision problems.

MacAskill writes:

[A]rguing that FDT does best in a class of ‘fair’ problems, without being able to define what that class is or why it’s interesting, is a pretty weak argument. And, even if we could define such a class of cases, claiming that FDT ‘appears to be superior’ to EDT and CDT in the classic cases in the literature is simply begging the question: CDT adherents claims that two-boxing is the right action (which gets you more expected utility!) in Newcomb’s problem; EDT adherents claims that smoking is the right action (which gets you more expected utility!) in the smoking lesion. The question is which of these accounts is the right way to understand ‘expected utility’; they’ll therefore all differ on which of them do better in terms of getting expected utility in these classic cases.

In my view, the epistemological issues are the most severe ones. I think Sharon Street’s paper A Darwinian Dilemma for Realist Theories of Value, for example, presents an especially hard-to-counter attack on the realist position on epistemological grounds. She argues that, in the light of the view that our brains evolved via natural selection, and natural selection did not and could not have directly selected for the accuracy of our normative intuitions, it is extremely difficult to construct a compelling explanation for why our normative intuitions should be correlated in any way with normative facts. This technically leave open the possibility of there being non-trivial normative facts, without us having any way of perceiving or intuiting them, but this state of affairs would strike most people as absurd. Although some realists, including Parfit, have attempted to counter Street’s argument, I’m not aware of anyone who I feel has truly succeeded. Street's argument pretty much just seems to work to me. ↩︎

These metaphysical and epistemological issues become less concerning if we accept some version of “naturalist realism” which asserts that all normative claims can be reduced into claims about the natural world (i.e. claims about physical and psychological properties) and therefore tested in roughly the same way we might test any other claim about the natural world. However, this view seems wrong to me.

The bluntest objection to naturalist realism is what's sometimes called the "just-too-different" objection. This is the objection that, to many and perhaps most people, normative claims are just obviously a different sort of claim. No one has ever felt any inclination to evoke an "is/is-made-of-wood divide" or an "is/is-illegal-in-Massachusetts divide," because the property of being made of wood and the propery of being illegal in Massachusetts are obviously properties of the standard (natural) kind. But references to the "is/ought divide" -- or, equivalently, the distinction between the "positive" and the "normative" -- are commonplace and don't typically provoke blank stares. Normative discussions are, seemingly, about something above-and-beyond and distinct from discussions of the physical and psychological aspects of a situation. When people debate whether or not it's "wrong" to support the death penalty or "wrong" for women to abort unwanted pregnancies, for example, it seems obvious that physical and psychological facts are typically not the core (or at least only) thing in dispute.

G.E. Moore’s “Open Question Argument" elaborates on this objection. The argument also raises the point that that, in many cases where we are inclined to ask “What should I do?”, it seems like what we are inclined to ask goes above-and-beyond any individual question we might ask about the natural world. Consider again the case where we are considering a career change and wondering what we should do. It seems like we could know all of the natural facts -- facts like how happy will we be on average while pursuing each career, how satisfied will we feel looking back on each career, how many lives we could improve by donating money made in each career, what labor practices each company has, how disappointed our parents will be if we pursue each career, how our personal values will change if we pursue each career, what we would end up deciding at the end of one hypothetical deliberative process or another, etc. -- and still retain the inclination to ask, “Given all this, what should I do?” This means that -- insofar as we're taking the realist stance that this question actually has a meaningful answer, rather than rejecting the question as vacuous -- the claim that we "should" do one thing or another cannot easily be understood as a claim about the natural world. A set of claims about the natural world may support the claim that we should make a certain decision, but, in cases such as this one, it seems like no set of claims about the natural world is equivalent to the claim that we should make a certain decision.

A last objection to mention is Parfit’s “Triviality Objection” (On What Matters, Section 95). The basic intuition behind Parfit’s objection is that pretty much any attempt to define the word “should” in terms of natural properties would turn many normative claims into puzzling assertions of either obvious tautologies or obvious falsehoods. For example, consider a man who is offered -- at the end of his life, I guess by the devil or something -- the option of undergoing a year of certain torture for a one-in-a-trillion chance of receiving a big prize: a trillion years of an equivalently powerful positive experience, plus a single lollipop. He is purely interested in experiencing pleasure and avoiding pain and would like to know whether he should take the offer. A decision theorist who endorses expected desire-fulfillment maximisation says that he “should,” since the lollipop tips the offer over into having slightly positive expected value. A decision theorist who endorses risk aversion says he “should not,” since the man is nearly certain to be horribly tortured without receiving any sort of compensation. In this context, it’s hard to understand how we could redefine the claim “He should take action X” in terms of natural properties and have this disagreement make any sense. We could define the phrase as meaning “Action X maximizes expected fulfillment of desire,” but now the first decision theorist is expressing an obvious tautology and the second decision theorist is expressing an obvious falsehood. We could also try, in keeping with a suggestion by Eliezer Yudkowsky, to define the phrase as meaning “Action X is the one that someone acting in a winning way would take.” But this is obviously too vague to imply a particular action; taking the gamble is associated with some chance of winning and some chance of losing. We could make the definition more specific -- for instance, saying “Action X is the one that someone acting in a way that maximizes expected winning would take” -- but now of course we’re back in tautology mode. The apparent upshot, here, is that many normative claims simply can’t be interpreted as non-trivially true or non-trivially false claims about natural properties. The associated disagreements only become sensible if we interpret them as being about something above-and-beyond these properties.

Of course, it is surely true that some of the claims people make using the word “should” can be understood as claims about the natural world. Words can, after all, be used in many different ways. But it’s the claims that can’t easily be understood in this way that non-naturalist realists such as Parfit, Enoch, and Moore have in mind. In general, I agree with the view that the key division in metaethics is between self-identified non-naturalist realists on the one hand and self-identified anti-realists and naturalist realists on the other hand, since “naturalist realists” are in fact anti-realists with regard to the distinctively normative properties of decisions that non-naturalist realists are talking about. If we rule out non-naturalist realism as a position then it seems the main remaining question is a somewhat boring one about semantics: When someone makes a statement of form “A should do X,” are they most commonly expressing some sort of attitude (non-cognitivism), making a claim about the natural world (naturalist realism), or making a claim about some made-up property that no actions actually possess (error theory)?

Here, for example, is how Michael Huemer (a non-naturalist realist) expresses this point in his book Ethical Intuitionism (pg. 8):

[Non-naturalist realists] differ fundamentally from everyone else in their view of the world. [Naturalist realists], non-cognitivists, and nihilists all agree in their basic view of the world, for they have no significant disagreements about what the non-evaluative facts are, and they all agree that there are no further facts over and above those. They agree, for example, on the non-evaluative properties of the act of stealing, and they agree, contra the [non-naturalist realists], that there is no further, distinctively evaluative property of the act. Then what sort of dispute do the [three] monistic theories have? I believe that, though this is not generally recognized, their disputes with each other are merely semantic. Once the nature of the world 'out there' has been agreed upon, semantic disputes are all that is left.

I think this attitude is in line with the viewpoint that Luke Muehlhauser expresses in his classic LessWrong blog post on what he calls “pluralistic moral reductionism.” PMR seems to me to be the view that: (a) non-naturalist realism is false, (b) all remaining meta-normative disputes are purely semantic, and (c) purely semantic disputes aren't terribly substantive and often reflect a failure to accept that the same phrase can be used in different ways. If we define the view this way, then, conditional on non-naturalist realism being false, I believe that PMR is the correct view. I believe that many non-naturalist realists would agree on this point as well. ↩︎

This point is made by Parfit in On What Matters. He writes: “We could not have decisive reasons to believe that there are no such normative truths, since the fact that we had these reasons would itself have to be one such truth. This point may not refute this kind of skepticism, since some skeptical arguments might succeed even if they undermined themselves. But this point shows how deep such skepticism goes, and how blank this skeptical state of mind would be” (On What Matters, Section 86). ↩︎

The PhilPapers survey suggests that philosophers who favor realism outweigh philosophers who favor anti-realism by about a 2:1 ratio. ↩︎