Epistemic Status: confused & unlikely

Author's note: the central claim of this article I now believe is confused, and mostly inaccurate. More precisely (in response to a comment by ChristianKl)

>Whose idea of reductionism are you criticising? I think your post could get more useful by being more clear about the idea you want to challenge.

I think this is closest I get to having a "Definiton 3.4.1" in my post

"...the other reductionism I mentioned, the 'big thing = small thing + small thing' one..."

Essentially, the claim is that to accurately explain reality, non-reductionist explanations aren't always *wrong*. 

The confusion, however, that I realized elsewhere in the thread, is that I conflate 'historical explanation' with 'predictive explanation'. Good predictive explanation will almost always be reductionist, because, as it says on the tin, big are made of smaller things. Good historical explanations, though, will be contra-reductionist, they'll explain phenomena in terms of its relation to the environment. Consider evolution; the genes seem to be explained non-reductionistically because their presence or absence is determined by its effect on the environment i.e. whether its fit, so the explanation for how it got there necessarily includes complex things because they cause it.

>Apart from that I don't know what you mean with theory in "Reductionism is a philosophy, not a theory." As a result on using a bunch of terms where I don't know exactly what you mean it's hard to follow your argument.

Artifact of confusion;  if contra-reductionism is a valid platform for explanation, then the value of reductionism isn't constative -- that is, it isn't about whether it's true or false, but something at the meta-level, rather than the object level. The antecedent is no longer believed, so now I do not believe the consequent.

The conceit I had by calling it a philosophy, or more accurately, a perspective, is essentially that you have a dataset, then you can apply a 'reductionist' filter on it to get reductionist explanations and a 'contra-reductionist' filter to get contra explanations. This was a confusion; and only seemed reasonable because I I was treating the two type of explanation -- historical and predictive -- as somehow equivalent, which I now know to be mistaken.

Reductionism is usually thought of as the assertion that the sum of the parts equal the whole. Or, a bit more polemically, that reductionist explanations more meaningful, proper, or [insert descriptor laced with postive affect]. It's certainly appealing, you could even say it seems reality prefers these types of explanation. The facts of biology can be attributed to the effects of chemistry, the reactions of chemistry can be attributed to the interplay of atoms, and so on.

But this is conflating what is seen with the perspective itself; I see a jelly donut therefore I am a jelly donut is not a valid inference. Reductionism is a way of thinking about facts, but it is not the facts themselves. Reductionism is a philosophy, not a theory. The closest thing to an testable prediction it makes it what could be termed an anti-prediction.

Another confusion concerns the alternatives to reductionism. The salient instance of anti-reduction tends to be some holist quantum spirituality woo, but I contend this is more of a weak man than anything. To alleviate any confusion, I'll just refer to my proposed notion as 'contra-reductionism'.

Earlier, I mentioned reductionism makes no meaningful predictions. To clarify this, I'll distinguish from a kind a diminutive motte of reductionism which may or may not actually exist outside my own mind, (and which truly is just a species of causality, broadly construed). In broad strokes, this reductionism 'reduces' a phenomena to the sum of it's causes, as opposed to its parts. This is the kind of reductionist explanation that treats evolution as a reductionist explanation, indeed it treats almost any model which isn't strictly random as 'reductionist'. The other referent would be reductionism as the belief that "big things are made of a smaller things, and complex things are made of simpler things". 

It's is the former kind of reductionism that makes what I labeled an anti-prediction, the core of this argument is simply that reductionist is about causality; specifically, it qualifies what types of causes should even be considered meaningful or well-founded or simply, worth thinking about. If you broaden the net sufficiently, causality is a concept which even makes sense to apply to mathematical abstraction completely unrooted in any kind of time. That is the interventionist account of causality essentially boils it down to 'what levers could we have pulled to make something not happen', which perfectly translates to maths, see, for instance, reductio ad absurdum arguments.

But I digress. This diminutive reductionism here is simply the belief that things can be reduced to their causes, which is on par with defining transhumanism as 'simplified humanism' in the category of useless philosophical mottes. In short, this is quite literally an assertion of no substance, and isn't even worth giving a name.

Now that I've finished attacking straw men, the other reductionism I mentioned, the 'big thing = small thing + small thing' one, is also flawed, albeit useful nonetheless.

This can be illustrated by the example of evolution I mentioned: An evolutionary explanation is actually anti-reductionist; it explains the placement of nucleotides in terms of mathematics like inclusive genetic fitness and complexities like population ecology. Put bluntly, the there is little object-level difference between explaining genes sequences with evolution and explaining weather with pantheons of gods (there is meta-level difference; i.e. one is accurate). Put less controversially, this is explicitly non-reductionistic; relatively simple things (the genetic sequence of a creature) are explained in the language of things far more complex (population and environment dynamics over the course of billions of years). If this is your reductionism, all it does is encapsulate the ontology of universe-space, or more evocatively, it's a logic that doesn't -- couldn't -- tell you where you live, because doesn't change wherever you may go.

Another situation where reductionism  and contra-reductionism give different answers is an example cribbed from David Deutsch. It's possible to set up dominos so that they compute an algorithm which decides the primality of 631. How would you explain a a positive result?

The reductionist explanation is approximately: "the domino remains standing because the one behind it didn't fall over", and so on with variation such as "that domino didn't fall over because the one behind it was knockovered sideways". The contra-reductionist explanation is "that domino didn't fall over "because 631 is prime". Each one is 'useful' depending on whether you are concerned with the mechanics of the domino computer or the theory.

You might detect something in these passages -- that while I slough off any pretense of reductionism, glorious (philosophical) materialism remains a kind of true north in my analysis. This is my thesis. My contra-reductionism isn't non-materialistic, it's merely a perspective inversion of the sort highlighted by a figure/ground illusion. Reductionism defines -- reduces -- objects by pointing to their constituents. A mechanism functions because its components function. A big thing of small things. Quasi-reductionism  does the opposite, it defined objects by their impact on other objects, "[A] tree is only a tree in the shade it gives to the ground below, to the relationship of wind to branch and air to leaf." I don't mean this in a spiritual way, naturally (no pun intended). I am merely defining objects externally rather than internally. At the core, the rose is still a rose, the sum is still normality.

If I had to give a short, pithy summation of this post, the core is simply that, like all systematized notions of truth or meaningfulness, reductionism collapses in degenerate cases where it fails to be useful or give the right answer. Contra-reductionism isn't a improvement or a replacement, but a alternative formulation in a conceptual monoculture, which happens to give right answer sometimes.

It seems like in order to go from P(H) to P(H|E) you have to become certain that E. Am I wrong about that? 

Say you have the following joint distribution:

P(H&E) = a
P(~H&E) = b
P(H&~E) = c

P(~H&~E) = d 

Where a,b,c, and d, are each larger than 0.

So P(H|E) = a/(a+b). It seems like what we're doing is going from assigning ~E some positive probability to assigning it a 0 probability. Is there another way to think about it? Is there something special about evidential statements that justifies changing their probabilities without having updated on something else? 

Edit: the below paragraphs are wrong. See the comments for an explanation.

Some people believe that the consciousness currently in one's body is the "same" consciousness as the one that was in one's body in the past and the one that will be in it in the future, but a "different" consciousness from those in other bodies. In this post I dissolve the question.

The question is meaningless because the answer doesn't correspond to any physical state in the universe and in no way influences or is influenced by sensory experiences. If one's consciousness suddenly became a totally different one, we know of no quantum particles that would change. Furthermore, swapping consciousnesses would make no changes to what is perceived. E.g. if one agent perceives p and time t and p' at the next time t+1, and another agent perceives q at time t and q' at time t+1, then if their consciousnesses are "swapped," the percepts would still be identical: p and q will be perceived at time t, and p' and q' will be perceived at t+1. One could argue that the percepts did change because the consciousness-swapping changed what a particular consciousness at time t will perceive at t+1, but that presupposes that a future consciousness will be in some meaningful way the "same" consciousness as the current one! Thus, the statement that two consciousnesses are the same consciousness is meaningless.

Can you find any flaws in my reasoning?

Standard methods of inferring knowledge about the world are based off premises that I don’t know the justifications for. Any justification (or a link to an article or book with one) for why these premises are true or should be assumed to be true would be appreciated.

Here are the premises:

• “One has knowledge of one’s own percepts.” Percepts are often given epistemic privileges, meaning that they need no justification to be known, but I see no justification for giving them epistemic privileges. It seems like the dark side of epistemology to me.

• “One’s reasoning is trustworthy.” If one’s reasoning is untrustworthy, then one’s evaluation of the trustworthiness of one’s reasoning can’t be trusted, so I don’t see how one could determine if one’s reasoning is correct. Why should one even consider one’s reasoning is correct to begin with? It seems like privileging the hypothesis, as there are many different ways one’s mind could work, and presumably only a very small proportion of possible minds would be remotely valid reasoners.

• “One’s memories are true.” Though one’s memories of how the world works gives a consistent explanation of why one is perceiving one’s current percepts, a perhaps simpler explanation is that the percepts one are currently experiencing are the only percepts one has ever experienced, and one’s memories are false. This hypothesis is still simple, as one only needs to have a very small number of memories, as one can only think of a small number of memories at any one time, and the memory of having other memories could be false as well.

A Computational Foundation for the Study of Cognition by David Chalmers

Abstract from the paper:

Computation is central to the foundations of modern cognitive science, but its role is controversial. Questions about computation abound: What is it for a physical system to implement a computation? Is computation sufficient for thought? What is the role of computation in a theory of cognition? What is the relation between different sorts of computational theory, such as connectionism and symbolic computation? In this paper I develop a systematic framework that addresses all of these questions.

Justifying the role of computation requires analysis of implementation, the nexus between abstract computations and concrete physical systems. I give such an analysis, based on the idea that a system implements a computation if the causal structure of the system mirrors the formal structure of the computation. This account can be used to justify the central commitments of artificial intelligence and computational cognitive science: the thesis of computational sufficiency, which holds that the right kind of computational structure suffices for the possession of a mind, and the thesis of computational explanation, which holds that computation provides a general framework for the explanation of cognitive processes. The theses are consequences of the facts that (a) computation can specify general patterns of causal organization, and (b) mentality is an organizational invariant, rooted in such patterns. Along the way I answer various challenges to the computationalist position, such as those put forward by Searle. I close by advocating a kind of minimal computationalism, compatible with a very wide variety of empirical approaches to the mind. This allows computation to serve as a true foundation for cognitive science.

See my welcome thread submission for a brief description of how I conceive of this as the first step towards formalizing friendliness.

In the not too distant past, people thought that our universe might be capable of supporting an unlimited amount of computation. Today our best guess at the cosmology of our universe is that it stops being able to support any kind of life or deliberate computation after a finite amount of time, during which only a finite amount of computation can be done (on the order of something like 10^120 operations).

Consider two hypothetical people, Tom, a total utilitarian with a near zero discount rate, and Eve, an egoist with a relatively high discount rate, a few years ago when they thought there was .5 probability the universe could support doing at least 3^^^3 ops and .5 probability the universe could only support 10^120 ops. (These numbers are obviously made up for convenience and illustration.) It would have been mutually beneficial for these two people to make a deal: if it turns out that the universe can only support 10^120 ops, then Tom will give everything he owns to Eve, which happens to be $1 million, but if it turns out the universe can support 3^^^3 ops, then Eve will give $100,000 to Tom. (This may seem like a lopsided deal, but Tom is happy to take it since the potential utility of a universe that can do 3^^^3 ops is so great for him that he really wants any additional resources he can get in order to help increase the probability of a positive Singularity in that universe.)

You and I are not total utilitarians or egoists, but instead are people with moral uncertainty. Nick Bostrom and Toby Ord proposed the Parliamentary Model for dealing with moral uncertainty, which works as follows:

Suppose that you have a set of mutually exclusive moral theories, and that you assign each of these some probability.  Now imagine that each of these theories gets to send some number of delegates to The Parliament.  The number of delegates each theory gets to send is proportional to the probability of the theory.  Then the delegates bargain with one another for support on various issues; and the Parliament reaches a decision by the delegates voting.  What you should do is act according to the decisions of this imaginary Parliament.

It occurred to me recently that in such a Parliament, the delegates would makes deals similar to the one between Tom and Eve above, where they would trade their votes/support in one kind of universe for votes/support in another kind of universe. If I had a Moral Parliament active back when I thought there was a good chance the universe could support unlimited computation, all the delegates that really care about astronomical waste would have traded away their votes in the kind of universe where we actually seem to live for votes in universes with a lot more potential astronomical waste. So today my Moral Parliament would be effectively controlled by delegates that care little about astronomical waste.

It's just struck me that there might be a tension between the coherence (C) and the extrapolated (E) part of CEV. One reason that CEV might work is that the mindspace of humanity isn't that large - humans are pretty close to each other, in comparison to the space of possible minds. But this is far more true in every day decisions than in large scale ones.

Take a fundamentalist Christian, a total utilitarian, a strong Marxist, an extreme libertarian, and a couple more stereotypes that fit your fancy. What can their ideology tell us about their everyday activities? Well, very little. Those people could be rude, polite, arrogant, compassionate, etc... and their ideology is a very weak indication of that. Different ideologies and moral systems seem to mandate almost identical everyday and personal interactions (this is in itself very interesting, and causes me to see many systems of moralities as formal justifications of what people/society find "moral" anyway).

But now let's more to a more distant - "far" - level. How will these people vote in elections? Will they donate to charity, and if so, which ones? If they were given power (via wealth or position in some political or other organisation), how are they likely to use that power? Now their ideology is much more informative. Though it's not fully determinative, we would start to question the label if their actions at this level seemed out of synch. A Marxist that donated to a Conservative party, for instance, would give us pause, and we'd want to understand the apparent contradiction.

Let's move up yet another level. How would they design or change the universe if they had complete power? What is their ideal plan for the long term? At this level, we're entirely in far mode, and we would expect that their vastly divergent ideologies would be the most informative piece of information about their moral preferences. Details about their character and personalities, which loomed so large at the everyday level, will now be of far lesser relevance. This is because their large scale ideals are not tempered by reality and by human interactions, but exist in a pristine state in their minds, changing little if at all. And in almost every case, the world they imagine as their paradise will be literal hell for the others (and quite possibly for themselves).

To summarise: the human mindspace is much narrower in near mode than in far mode.

And what about CEV? Well, CEV is what we would be "if we knew more, thought faster, were more the people we wished we were, had grown up farther together". The "were more the people we wished we were" is going to be dominated by the highly divergent far mode thinking. The "had grown up farther together" clause attempts to mesh these divergences, but that simply obscures the difficulty involved. The more we extrapolate, the harder coherence becomes.

It strikes me that there is a strong order-of-operations issue here. I'm not a fan of CEV, but it seems it would be much better to construct, first, the coherent volition of humanity, and only then to extrapolate it.

Consequentialism traditionally doesn't distinguish between acts of commission or acts of omission. Not flipping the lever to the left is equivalent with flipping it to the right.

But there seems one clear case where the distinction is important. Consider a moral learning agent. It must act in accordance with human morality and desires, which it is currently unclear about.

For example, it may consider whether to forcibly wirehead everyone. If it does so, they everyone will agree, for the rest of their existence, that the wireheading was the right thing to do. Therefore across the whole future span of human preferences, humans agree that wireheading was correct, apart from a very brief period of objection in the immediate future. Given that human preferences are known to be inconsistent, this seems to imply that forcible wireheading is the right thing to do (if you happen to personally approve of forcible wireheading, replace that example with some other forcible rewriting of human preferences).

What went wrong there? Well, this doesn't respect "conversation of moral evidence": the AI got the moral values it wanted, but only though the actions it took. This is very close to the omission/commission distinction. We'd want the AI to not take actions (commission) that determines the (expectation of the) moral evidence it gets. Instead, we'd want the moral evidence to accrue "naturally", without interference and manipulation from the AI (omission).

In this post, I list six metaethical possibilities that I think are plausible, along with some arguments or plausible stories about how/why they might be true, where that's not obvious. A lot of people seem fairly certain in their metaethical views, but I'm not and I want to convey my uncertainty as well as some of the reasons for it.

1. Most intelligent beings in the multiverse share similar preferences. This came about because there are facts about what preferences one should have, just like there exist facts about what decision theory one should use or what prior one should have, and species that manage to build intergalactic civilizations (or the equivalent in other universes) tend to discover all of these facts. There are occasional paperclip maximizers that arise, but they are a relatively minor presence or tend to be taken over by more sophisticated minds.
2. Facts about what everyone should value exist, and most intelligent beings have a part of their mind that can discover moral facts and find them motivating, but those parts don't have full control over their actions. These beings eventually build or become rational agents with values that represent compromises between different parts of their minds, so most intelligent beings end up having shared moral values along with idiosyncratic values.
3. There aren't facts about what everyone should value, but there are facts about how to translate non-preferences (e.g., emotions, drives, fuzzy moral intuitions, circular preferences, non-consequentialist values, etc.) into preferences. These facts may include, for example, what is the right way to deal with ontological crises. The existence of such facts seems plausible because if there were facts about what is rational (which seems likely) but no facts about how to become rational, that would seem like a strange state of affairs.
4. None of the above facts exist, so the only way to become or build a rational agent is to just think about what preferences you want your future self or your agent to hold, until you make up your mind in some way that depends on your psychology. But at least this process of reflection is convergent at the individual level so each person can reasonably call the preferences that they endorse after reaching reflective equilibrium their morality or real values.
5. None of the above facts exist, and reflecting on what one wants turns out to be a divergent process (e.g., it's highly sensitive to initial conditions, like whether or not you drank a cup of coffee before you started, or to the order in which you happen to encounter philosophical arguments). There are still facts about rationality, so at least agents that are already rational can call their utility functions (or the equivalent of utility functions in whatever decision theory ends up being the right one) their real values.
6. There aren't any normative facts at all, including facts about what is rational. For example, it turns out there is no one decision theory that does better than every other decision theory in every situation, and there is no obvious or widely-agreed-upon way to determine which one "wins" overall.

(Note that for the purposes of this post, I'm concentrating on morality in the axiological sense (what one should value) rather than in the sense of cooperation and compromise. So alternative 1, for example, is not intended to include the possibility that most intelligent beings end up merging their preferences through some kind of grand acausal bargain.)

It may be useful to classify these possibilities using labels from academic philosophy. Here's my attempt: 1. realist + internalist 2. realist + externalist 3. relativist 4. subjectivist 5. moral anti-realist 6. normative anti-realist. (A lot of debates in metaethics concern the meaning of ordinary moral language, for example whether they refer to facts or merely express attitudes. I mostly ignore such debates in the above list, because it's not clear what implications they have for the questions that I care about.)

One question LWers may have is, where does Eliezer's metathics fall into this schema? Eliezer says that there are moral facts about what values every intelligence in the multiverse should have, but only humans are likely to discover these facts and be motivated by them. To me, Eliezer's use of language is counterintuitive, and since it seems plausible that there are facts about what everyone should value (or how each person should translate their non-preferences into preferences) that most intelligent beings can discover and be at least somewhat motivated by, I'm reserving the phrase "moral facts" for these. In my language, I think 3 or maybe 4 is probably closest to Eliezer's position.

Basically Heather Dyke argues that metaphysicians are too often arguing from representations of reality (eg in language) to reality itself.

 It looks to me like a variant of the mind projection fallacy. This might be the first book length treatment teh fallacy has gotten though.  What do people think?

See reviews here

https://www.sendspace.com/file/k5x8sy

https://ndpr.nd.edu/news/23820-metaphysics-and-the-representational-fallacy/

To give bit of background there's a debate between A-theorists and B-theorists in philosophy of time.

A-theorists think time has ontological distinctions between past present and future

B-theorists hold there is no ontological distinction between past present and future.

Dyke argues that a popular argument for A-theory (tensed language represents ontological distinctions) commits the representational fallacy. Bourne agrees , but points out an argument Dyke uses for B-theory commits the same fallacy.

Consider some disputed proposition X. Suppose there appeared to be a limited number of ways of proving and of disproving X. No one has yet constructed a proof or disproof, but you have a feeling for how likely it is that someone will.

For instance, take Fermat's Last Theorem or the 4-color problem. For each of them, at one point in time, there was no proof, but people had some sense of the prior probability of observing the lack of a counterexample given the space searched so far. They could use that to assign a probability of there being a counterexample (and hence a disproof) [1]. Later, there was an alleged proof, and people could estimate the probability that the proof was correct based on the reputation of the prover and the approach used. At that point, people could assign values to both P(will_be_proven(X)) and P(will_be_disproven(X)).

Is it reasonable to assign P(X) = P(will_be_proven(X)) / (P(will_be_proven(X)) + P(will_be_disproven(X))) ?

If so, consider X = "free will exists". One could argue that the term "free will" is defined such that it is impossible to detect it, or to prove that it exists. But if one could prove that the many worlds interpretation of quantum mechanics is correct, that would constitute a disproof of X. Then P(will_be_proven(X)) / (P(will_be_proven(X)) + P(will_be_disproven(X))) = 0.

Is it possible for this to happen when you know that X is not undecidable? If so, what do you do then?

1. The computation is not as simple as it might appear, because you need to adjust for the selection effect of mathematicians being interested only in conjectures with no counterexamples.

I am about to graduate from one of the only universities in the world that has a high concentration of high-caliber analytic philosophers who are theists. (Specifically, the University of Notre Dame, IN) So as not to miss this once-in-a-lifetime opportunity, I have sent out emails asking many of them if they would like to meet and discuss their theism with me. Several of them have responded already in the affirmative; fingers crossed for the rest. I'm really looking forward to this because these people are really smart, and have spent a lot of time thinking about this, so I expect them to have interesting and insightful things to say.

Do you have suggestions for questions I could ask them? My main question will of course be "Why do you believe in God?" and variants thereof, but it would be nice if I could say e.g. "How do you avoid the problem of X which is a major argument against theism?"

Questions I've already thought of:

1-Why do you believe in God?

2-What are the main arguments in favor of theism, in your opinion?

3-What about the problem of evil? What about objective morality: how do you make sense of it, and if you don't, then how do you justify God?

4-What about divine hiddenness? Why doesn't God make himself more easily known to us? For example, he could regularly send angels to deliver philosophical proofs on stone tablets to doubters.

5-How do you explain God's necessary existence? What about the "problem of many Gods," i.e. why can't people say the same thing about a slightly different version of God?

6-In what sense is God the fundamental entity, the uncaused cause, etc.? How do you square this with God's seeming complexity? (he is intelligent, after all) If minds are in fact simple, then how is that supposed to work?

I welcome more articulate reformulations of the above, as well as completely new ideas.

Cross-posted from the MIRI blog.

For centuries, philosophers wondered how we could learn what causes what. Some argued it was impossible, or possible only via experiment. Others kept hacking away at the problem, clarifying ideas like counterfactual and probability and correlation by making them more precise and coherent.

Then, in the 1990s, a breakthrough: Judea Pearl and others showed that, in principle, we can sometimes infer causal relations from data even without experiment, via the mathematical machinery of probabilistic graphical models.

Next, engineers used this mathematical insight to write software that can, in seconds, infer causal relations from a data set of observations.

Across the centuries, researchers had toiled away, pushing our understanding of causality from philosophy to math to engineering.

And so it is with Friendly AI research. Current progress on each sub-problem of Friendly AI lies somewhere on a spectrum from philosophy to math to engineering.

We began with some fuzzy philosophical ideas of what we want from a Friendly AI (FAI). We want it to be benevolent and powerful enough to eliminate suffering, protect us from natural catastrophes, help us explore the universe, and otherwise make life awesome. We want FAI to allow for moral progress, rather than immediately reshape the galaxy according to whatever our current values happen to be. We want FAI to remain beneficent even as it rewrites its core algorithms to become smarter and smarter. And so on.

Small pieces of this philosophical puzzle have been broken off and turned into math, e.g. Pearlian causal analysis and Solomonoff induction. Pearl's math has since been used to produce causal inference software that can be run on today's computers, whereas engineers have thus far succeeded in implementing (tractable approximations of) Solomonoff induction only for very limited applications.

Toy versions of two pieces of the "stable self-modification" problem were transformed into math problems in de Blanc (2011) and Yudkowsky & Herreshoff (2013), though this was done to enable further insight via formal analysis, not to assert that these small pieces of the philosophical problem had been solved to the level of math.

Thanks to Patrick LaVictoire and other MIRI workshop participants,¹ Douglas Hofstadter's FAI-relevant philosophical idea of "superrationality" seems to have been, for the most part, successfully transformed into math, and a bit of the engineering work has also been done.

I say "seems" because, while humans are fairly skilled at turning math into feats of practical engineering, we seem to be much less skilled at turning philosophy into math, without leaving anything out. For example, some very sophisticated thinkers have claimed that "Solomonoff induction solves the problem of inductive inference," or that "Solomonoff has successfully invented a perfect theory of induction." And indeed, it certainly seems like a truly universal induction procedure. However, it turns out that Solomonoff induction doesn't fully solve the problem of inductive inference, for relatively subtle reasons.²

Unfortunately, philosophical mistakes like this could be fatal when humanity builds the first self-improving AGI (Yudkowsky 2008).³ FAI-relevant philosophical work is, as Nick Bostrom says, "philosophy with a deadline."

¹ And before them, Moshe Tennenholtz.

² Yudkowsky plans to write more about how to improve on Solomonoff induction, later.

³ This is a specific instance of a problem Peter Ludlow described like this: "the technological curve is pulling away from the philosophy curve very rapidly and is about to leave it completely behind."

EDIT: I've found a very relevant indicator for my question, see "Quality of life" criteria below.

My main question is: which non-academic factors should I consider when moving to another country/city for a PhD? Further, I would also like to evaluate each country/city¹ according to those criteria, but first I need to know which are the relevant criteria. If you know any (any at all) scientific literature on moving to another country and well being, let me know.

I've lived in Brazil all my life, I really like it here for many reasons. Mostly, by how personal relationships are established and maintained. However, Brazil's inability to construct a stable well developed society have crippled my intellectual development, and I simply cannot take it anymore - my brain will die here. Moreover, I feel like most of my high level desires(values) are much more in line with countries on the other end of the World Values Survey graphic. I have rational/secular and self-expressing values, instead of traditional-survival oriented ones. For all those reasons, I will be applying for my PhD aboard. I have pondered many of the career and academic factors involved, and I've had the help of many good and objective indexes available (e.g.: here and here). I've mapped most of the Departments of Philosophy in which I could research my topic (moral enhancement), and I believe these are the major factors. However, there is one other important factor I'm a bit clueless about: which country/city is better in all other aspects already not accounted by academic criteria?

My main options are²:

• 1st: Oxford (no need to explain)
• 2nd: Manchester (it's near Oxford, John Harris is there, one of the foremost researchers on moral enhancement)
• 3rd: Stockholm (where everyone is born a transhumanist)
• 3rd: Wellington, New Zealand (Nicholas Agar is there, one of the foremost researchers on moral enhancement)
• 4th: Some places in continental Europe I'm still investigating (e.g.: Zurich , Munich)
• 4th: Brazil (bioethics program in Rio de Janeiro)

However, this list is solely based on academic criteria. I need to factor in non-academic criteria. In fact, I do not even know which are the relevant non-academic criteria. That would be my first question.  I got fixated on the World Values Survey factors, but I might be wrong. I would gather the happiness index is important, but it might not vary for the same individual between countries, or it might covary oddly with the happiness index of the destination country.  My second question would be how each country/city is ranked according to these criteria.

There are many things that will be affected by accessing these other factors. First, I think Oxford is far, far above the 2nd option. But it is above enough that if I do not get in there on the first time (80% probability), I should wait and apply next year again instead of going to somewhere else where I did get accepted? Second, my current plan is to build the strongest possible application for Oxford and use it elsewhere. But if Oxford is not so clearly the undisputed 1st place, then I should be more concerned with building a good application that also accounted for other countries specific criteria. Furthermore, right now, I think I have a major bias against New Zealand. In terms of moral enhancement research it would be the second best after Oxford, it has huge human development, freedom and happiness indexes. However, the fact it is in the freaking middle of nowhere is very discouraging. Am I wrong about this? What are the correct factors I should be accounting for?

Here is a list of the factors I could gather from the comments, mostly the one by MathiasZaman:

• World Values Survey: Already explained above, I believe is one of the most important. But I wonder if I'm not biased and fixated on this. I would also like to have a Cities Values Survey, since in reality I'm choosing cities.

• Quality of life: It should matter. But I haven't found a good index for not-huge cities. The index for countries are well know. Sweden and New Zealand take the lead, then England and after a while Brazil. However, obviously, being an expatriate changes things a lot. If you know of an expatriates' quality of life index for cities or countries, please, let me know. However, there's one good indicator for expatriates available, but it is only for countries though.

• Happiness: It should matter. Or, it might not vary for the same individual between countries. I don't know. It is more or less the same as for quality of life, since it is a major component of it.

• Relative closeness to other countries: I'm having a hard time spelling out this one, but check this comment by Kaj.

• Language barrier: This is hard to account for. I'm expecting that in no developed country I would be put in a situation where relevant people (from my university) would not be talking in English if I'm on the conversation. If it is not true, this is majorly relevant. If it is true, this is mildly relevant. I would expect this would be both a function of English proficiency and willingness to talk in English. Note Sweden is the highest in proficiency and the rest of continental Europe is the lowest. However, I do not know how to find the "willingness" factor.

• Socio-economic system: Highly relevant. I believe this is accounted for on the World Values Survey, as type of government strongly covaries with values. More modern (rational-secular/self-expressing) have more liberal systems, while less modern have more strong governments. (while the really ancient ones have almost no State).

• Public transport and real estate: Highly practical and I would not have thought if not for the comments. Commuting times and cost are very important. Real estate also, one of the many reasons I have not considered London was because of extremely high rents. Also, this brings back to mind why I posted this. I remember reading a very useful post on how to choose a house, where it pointed out to many relevant but unaccounted factors, commuting was one of them. What I want is something similar for cities.

• Finances: It is mildly relevant, I do not believe I will have a desire for anything else besides researching, specially in Oxford. But I might be wrong. How I will finance myself is still a bit uncertain. For high ranking universities I will probably have a scholarship from Brazil, otherwise I will need a scholarship from elsewhere. With the probabilities in brackets, and some living costs factored in:
  • Oxford: Brazilian government scholarship. They will give me 1100 EUR per month besides paying for all the fees and accommodation. They pay one international travel per year. (90%) High living costs.
  • Manchester, same as above. (70%)
  • Stockholm: Swedish government salary (there a PhD is a job). For an Physics position it was ~2500 EUR per month.(100%) It has a very high living cost for expatriates
  • Wellington: I don't know, but will find out.
  • Brazil: 950 EUR per month (70%). Low living costs. 
• International status: Makes a huge difference if one lives in a city by desire or by merely being born there. Prima facie, one should be more interesting if she is there by desire. Thus, I should give priority to more international cities. I will have to use anecdotal evidence here, since on normal datasets low skilled immigrants will dominate the sample. If I were less busy, I would compile data on an university-by-university basis.

Finally, please remember this not a competition between countries or cities and refrain for expressing any, however tiny, nationalism on the comments. I'm not expressing my subjective feelings either, I'm merely trying to find out the relevant factors and how countries or cities rank according to them.

Footnotes:

1. I would mostly like to be comparing cities, which was what I did when accounting for academic criteria, however (a) some datas are only available for countries, (b) in some cases I do not know to which city I will go and (c) this makes the analysis more complex.

2. US is out of the table for 4 reasons: (1) I would have to throw my MPhil on the garbage and start over. (2) Isn't that far away from a survival-traditional oriented society. (3) GRE (philosophy is the most competitive PhD program, I would have to nearly ace it, and I simply can't do that at the present time) (4) Doesn't have many transhumanistic oriented philosophy departments, specially on the top universities. Canada is out for (1), (3) and (4).

Thought of this after reading the discussion following abcd_z's post on utilitarianism, but it seemed sufficiently different that I figured I'd post it as a separate topic. It feels like the sort of thing that must have been discussed on this site before, but I haven't seen anything like it (I don't really follow the ethical philosophy discussions here), so pointers to relevant discussion would be appreciated.

Let's say I start off with some arbitrary utility function and I have the ability to arbitrarily modify my own utility function. I then become convinced of the truth of preference utilitarianism. Now, presumably my new moral theory prescribes certain terminal values that differ from the ones I currently hold. To be specific, my moral theory tells me to construct a new utility function using some sort of aggregating procedure that takes as input the current utility functions of all moral agents (including my own). This is just a way of capturing the notion that if preference utilitarianism is true, then my behavior shouldn't be directed towards the fulfilment of my own (prior) goals, but towards the maximization of preference satisfaction. Effectively, I should self-modify to have new goals.

But once I've done this, my own utility function has changed, so as a good preference utilitarian, I should run the entire process over again, this time using my new utility function as one of the inputs. And then again, and again... Let's look at a toy model. In this universe, there are two people: me (a preference utilitarian) and Alice (not a preference utilitarian). Let's suppose Alice does not alter her utility function in response to changes in mine. There are two exclusive states of affairs that can be brought about in this universe: A and B. Alice assigns a utility of 10 to A and 5 to B, I initially assign a utility of 3 to A and 6 to B. Assuming the correct way to aggregate utility is by averaging, I should modify my utilities to 6.5 for A and 5.5 for B. Once I have done this, I should again modify to 8.25 for A and 5.25 for B. Evidently, my utility function will converge towards Alice's.

I haven't thought about this at all, but I think the same convergence will occur if we add more utilitarians to the universe. If we add more Alice-type non-utilitarians there is no guarantee of convergence. So anyway, this seems to me a pretty strong argument against utilitarianism. If we have a society of perfect utilitarians, a single defector who refuses to change her utility function in response to changes in others' can essentially bend the society to her will, forcing (through the power of moral obligation!) everybody else to modify their utility functions to match hers, no matter what her preferences actually are. Even if there are no defectors, all the utilitarians will self-modify until they arrive at some bland (value judgment alert) middle ground.

Now that I think about it, I suspect this is basically just a half-baked corollary to Bernard Williams' famous objection to utilitarianism:

The point is that [the agent] is identified with his actions as flowing from projects or attitudes which… he takes seriously at the deepest level, as what his life is about… It is absurd to demand of such a man, when the sums come in from the utility network which the projects of others have in part determined, that he should just step aside from his own project and decision and acknowledge the decision which utilitarian calculation requires. It is to alienate him in a real sense from his actions and the source of his action in his own convictions. It is to make him into a channel between the input of everyone's projects, including his own, and an output of optimific decision; but this is to neglect the extent to which his projects and his decisions have to be seen as the actions and decisions which flow from the projects and attitudes with which he is most closely identified. It is thus, in the most literal sense, an attack on his integrity.

Anyway, I'm sure ideas of this sort have been developed much more carefully and seriously by philosophers, or even other posters here at LW. As I said, any references would be greatly appreciated.

http://kajsotala.fi/2013/07/book-review-mindmelding-consciousness-neuroscience-and-the-minds-privacy/

I review William Hirstein's book Mindmelding: Consciousness, Neuroscience, and the Mind’s Privacy, which he proposes a way of connecting the brains of two different people together so that when person A has a conscious experience, person B may also have the same experience. In particular, I compare it to my and Harri Valpola's earlier paper Coalescing Minds, in which we argued that it would be possible to join the brains of two people together in such a way that they'd become a single mind.

Fortunately, it turns out that the book and the paper are actually rather nicely complementary. To briefly summarize the main differences, we intentionally skimmed over many neuroscientific details in order to establish mindmelding as a possible future trend, while Hirstein extensively covers the neuroscience but is mostly interested in mindmelding as a thought experiment. We seek to predict a possible future trend, while Hirstein seeks to argue a philosophical position: Hirstein focuses on philosophical implications while we focus on societal implications. Hirstein talks extensively about the possibility of one person perceiving another’s mental states while both remaining distinct individuals, while we mainly discuss the possibility of two distinct individuals coalescing together into one.

I expect that LW readers might be particularly interested in some of the possible implications of Hirstein's argument, which he himself didn't discuss in the book, but which I speculated on in the review:

Most obviously, if another person’s conscious states could be recorded and replayed, it would open the doors for using this as entertainment. Were it the case that you couldn’t just record and replay anyone’s conscious experience, but learning to correctly interpret the data from another brain would require time and practice, then individual method actors capable of immersing themselves in a wide variety of emotional states might become the new movie stars. Once your brain learned to interpret their conscious states, you could follow them in a wide variety of movie-equivalents, with new actors being hampered by the fact that learning to interpret the conscious states of someone who had only appeared in one or two productions wouldn’t be worth the effort. If mind uploading was available, this might give considerable power to a copy clan consisting of copies of the same actor, each participating in different productions but each having a similar enough brain that learning to interpret one’s conscious states would be enough to give access to the conscious states of all the others.

The ability to perceive various drug- or meditation-induced states of altered consciousness while still having one’s executive processes unhindered and functional would probably be fascinating for consciousness researchers and the general public alike. At the same time, the ability for anyone to experience happiness or pleasure by just replaying another person’s experience of it might finally bring wireheading within easy reach, with all the dangers associated with that.

A Hirstein-style mind meld might possibly also be used as an uploading technique. Some upload proposals suggest compiling a rich database of information about a specific person, and then later using that information to construct a virtual mind whose behavior would be consistent with the information about that person. While creating such a mind based on just behavioral data makes questionable the extent to which the new person would really be a copy of the original, the skeptical argument loses some of its force if we can also include in the data a recording of all the original’s conscious states during various points in their life. If we are able to use the data to construct a mind that would react to the same sensory inputs with the same conscious states as the original did, whose executive processes would manipulate those states in the same ways as the original, and who would take the same actions as the original did, would that mind then not essentially be the same mind as the original mind?

Hirstein’s argumentation is also relevant for our speculations concerning the evolution of mind coalescences. We spoke abstractly about the ”preferences” of a mind, suggesting that it might be possible for one mind to extract the knowledge from another mind without inherting its preferences, and noting that conflicting preferences would be one reason for two minds to avoid coalescing together. However, we did not say much about where in the brain preferences are produced, and what would be actually required for e.g. one mind to extract another’s knowledge without also acquiring its preferences. As the above discussion hopefully shows, some of our preferences are implicit in our automatic habits (the things that we show we value with our daily routines), some in the preprocessing of sensory data that our brains carry out (the things and ideas that are ”painted with” positive associations or feelings), and some in the configuration of our executive processes (the actions we actually end up doing in response to novel or conflicting situations). (See also.) This kind of a breakdown seems like very promising material for some neuroscience-aware philosopher to tackle in an attempt to figure out just what exactly preferences are; maybe someone has already done so.

I put "Friendliness" in quotes in the title, because I think what we really want, and what MIRI seems to be working towards, is closer to "optimality": create an AI that minimizes the expected amount of astronomical waste. In what follows I will continue to use "Friendly AI" to denote such an AI since that's the established convention.

I've often stated my objections MIRI's plan to build an FAI directly (instead of after human intelligence has been substantially enhanced). But it's not because, as some have suggested while criticizing MIRI's FAI work, that we can't foresee what problems need to be solved. I think it's because we can largely foresee what kinds of problems need to be solved to build an FAI, but they all look superhumanly difficult, either due to their inherent difficulty, or the lack of opportunity for "trial and error", or both.

When people say they don't know what problems need to be solved, they may be mostly talking about "AI safety" rather than "Friendly AI". If you think in terms of "AI safety" (i.e., making sure some particular AI doesn't cause a disaster) then that does looks like a problem that depends on what kind of AI people will build. "Friendly AI" on the other hand is really a very different problem, where we're trying to figure out what kind of AI to build in order to minimize astronomical waste. I suspect this may explain the apparent disagreement, but I'm not sure. I'm hoping that explaining my own position more clearly will help figure out whether there is a real disagreement, and what's causing it.

The basic issue I see is that there is a large number of serious philosophical problems facing an AI that is meant to take over the universe in order to minimize astronomical waste. The AI needs a full solution to moral philosophy to know which configurations of particles/fields (or perhaps which dynamical processes) are most valuable and which are not. Moral philosophy in turn seems to have dependencies on the philosophy of mind, consciousness, metaphysics, aesthetics, and other areas. The FAI also needs solutions to many problems in decision theory, epistemology, and the philosophy of mathematics, in order to not be stuck with making wrong or suboptimal decisions for eternity. These essentially cover all the major areas of philosophy.

For an FAI builder, there are three ways to deal with the presence of these open philosophical problems, as far as I can see. (There may be other ways for the future to turns out well without the AI builders making any special effort, for example if being philosophical is just a natural attractor for any superintelligence, but I don't see any way to be confident of this ahead of time.) I'll name them for convenient reference, but keep in mind that an actual design may use a mixture of approaches.

1. Normative AI - Solve all of the philosophical problems ahead of time, and code the solutions into the AI.
2. Black-Box Metaphilosophical AI - Program the AI to use the minds of one or more human philosophers as a black box to help it solve philosophical problems, without the AI builders understanding what "doing philosophy" actually is.
3. White-Box Metaphilosophical AI - Understand the nature of philosophy well enough to specify "doing philosophy" as an algorithm and code it into the AI.

The problem with Normative AI, besides the obvious inherent difficulty (as evidenced by the slow progress of human philosophers after decades, sometimes centuries of work), is that it requires us to anticipate all of the philosophical problems the AI might encounter in the future, from now until the end of the universe. We can certainly foresee some of these, like the problems associated with agents being copyable, or the AI radically changing its ontology of the world, but what might we be missing?

Black-Box Metaphilosophical AI is also risky, because it's hard to test/debug something that you don't understand. Besides that general concern, designs in this category (such as Paul Christiano's take on indirect normativity) seem to require that the AI achieve superhuman levels of optimizing power before being able to solve its philosophical problems, which seems to mean that a) there's no way to test them in a safe manner, and b) it's unclear why such an AI won't cause disaster in the time period before it achieves philosophical competence.

White-Box Metaphilosophical AI may be the most promising approach. There is no strong empirical evidence that solving metaphilosophy is superhumanly difficult, simply because not many people have attempted to solve it. But I don't think that a reasonable prior combined with what evidence we do have (i.e., absence of visible progress or clear hints as to how to proceed) gives much hope for optimism either.

To recap, I think we can largely already see what kinds of problems must be solved in order to build a superintelligent AI that will minimize astronomical waste while colonizing the universe, and it looks like they probably can't be solved correctly with high confidence until humans become significantly smarter than we are now. I think I understand why some people disagree with me (e.g., Eliezer thinks these problems just aren't that hard, relative to his abilities), but I'm not sure why some others say that we don't yet know what the problems will be.

Follow-up to: Normative uncertainty in Newcomb's problem

Philosophers and atheists break for two-boxing; theists and Less Wrong break for one-boxing
Personally, I would one-box on Newcomb's Problem. Conditional on one-boxing for lawful reasons, one boxing earns $1,000,000, while two-boxing, conditional on two-boxing for lawful reasons, would deliver only a thousand. But this seems to be firmly a minority view in philosophy, and numerous heuristics about expert opinion suggest that I should re-examine the view.

In the PhilPapers survey, Philosophy undergraduates start off divided roughly evenly between one-boxing and two-boxing:

Newcomb's problem: one box or two boxes?

But philosophy faculty, who have learned more (less likely to have no opinion), and been subject to further selection, break in favor of two-boxing:

Newcomb's problem: one box or two boxes?

Specialists in decision theory (who are also more atheistic, more compatibilist about free will, and more physicalist than faculty in general) are even more convinced:

Newcomb's problem: one box or two boxes?

Looking at the correlates of answers about Newcomb's problem, two-boxers are more likely to believe in physicalism about consciousness, atheism about religion, and other positions generally popular around here (which are also usually, but not always, in the direction of philosophical opinion). Zooming in one correlate, most theists with an opinion are one-boxers, while atheists break for two-boxing:

Less Wrong breaks overwhelmingly for one-boxing in survey answers for 2012:

NEWCOMB'S PROBLEM
One-box: 726, 61.4%
Two-box: 78, 6.6%
Not sure: 53, 4.5%
Don't understand: 86, 7.3%
No answer: 240, 20.3%

When I elicited LW confidence levels in a poll, a majority indicated 99%+ confidence in one-boxing, and 77% of respondents indicated 80%+ confidence.

What's going on?

I would like to understand what is driving this difference of opinion. My poll was a (weak) test of the hypothesis that Less Wrongers were more likely to account for uncertainty about decision theory: since on the standard Newcomb's problem one-boxers get $1,000,000, while two-boxers get $1,000, even a modest credence in the correct theory recommending one-boxing could justify the action of one-boxing.

If new graduate students read the computer science literature on program equilibrium, including some local contributions like Robust Cooperation in the Prisoner's Dilemma and A Comparison of Decision Algorithms on Newcomblike Problems, I would guess they would tend to shift more towards one-boxing. Thinking about what sort of decision algorithms it is rational to program, or what decision algorithms would prosper over numerous one-shot Prisoner's Dilemmas with visible source code, could also shift intuitions. A number of philosophers I have spoken with have indicated that frameworks like the use of causal models with nodes for logical uncertainty are meaningful contributions to thinking about decision theory. However, I doubt that for those with opinions, the balance would swing from almost 3:1 for two-boxing to 9:1 for one-boxing, even concentrating on new decision theory graduate students.

On the other hand, there may be an effect of unbalanced presentation to non-experts. Less Wrong is on average less philosophically sophisticated than professional philosophers. Since philosophical training is associated with a shift towards two-boxing, some of the difference in opinion could reflect a difference in training. Then, postings on decision theory have almost all either argued for or assumed one-boxing as the correct response on Newcomb's problem. It might be that if academic decision theorists were making arguments for two-boxing here, or if there was a reduction in pro one-boxing social pressure, there would be a shift in Less Wrong opinion towards two-boxing.

Less Wrongers, what's going on here? What are the relative causal roles of these and other factors in this divergence?

ETA: The SEP article on Causal Decision Theory.

Here is Wikipedia's description of Newcomb's problem:

The player of the game is presented with two boxes, one transparent (labeled A) and the other opaque (labeled B). The player is permitted to take the contents of both boxes, or just the opaque box B. Box A contains a visible $1,000. The contents of box B, however, are determined as follows: At some point before the start of the game, the Predictor makes a prediction as to whether the player of the game will take just box B, or both boxes. If the Predictor predicts that both boxes will be taken, then box B will contain nothing. If the Predictor predicts that only box B will be taken, then box B will contain $1,000,000.

Nozick also stipulates that if the Predictor predicts that the player will choose randomly, then box B will contain nothing.

By the time the game begins, and the player is called upon to choose which boxes to take, the prediction has already been made, and the contents of box B have already been determined. That is, box B contains either $0 or $1,000,000 before the game begins, and once the game begins even the Predictor is powerless to change the contents of the boxes. Before the game begins, the player is aware of all the rules of the game, including the two possible contents of box B, the fact that its contents are based on the Predictor's prediction, and knowledge of the Predictor's infallibility. The only information withheld from the player is what prediction the Predictor made, and thus what the contents of box B are.

Most of this is a fairly general thought experiment for thinking about different decision theories, but one element stands out as particularly arbitrary: the ratio between the amount the Predictor may place in box B and the amount in box A. In the Newcomb formulation conveyed by Nozick, this ratio is 1000:1, but this is not necessary. Most decision theories that recommend one-boxing do so as long as the ratio is greater than 1.

The 1000:1 ratio strengthens the intuition for one-boxing, which is helpful for illustrating why one might find one-boxing plausible. However, given uncertainty about normative decision theory, the decision to one-box can diverge from one's best guess at the best decision theory, e.g. if I think there is a 1 in 10 chance that one-boxing decision theories I may one-box on Newcomb's problem with a potential payoff ratio of 1000:1 but not if the ratio is only 2:1.

So the question, "would you one-box on Newcomb's problem, given your current state of uncertainty?" is not quite the same as "would the best decision theory recommend one-boxing?" This occurred to me in the context of this distribution of answers among target philosophy faculty from the PhilPapers Survey:

Newcomb's problem: one box or two boxes?

If all of these answers are about the correct decision theory (rather than what to do in the actual scenario), then two-boxing is the clear leader, with a 2.85:1 ratio of support (accept or lean) in its favor, but this skew would seem far short of that needed to justify 1000:1 confidence in two-boxing on Newcomb's Problem.

Here are Less Wrong survey answers for 2012:

NEWCOMB'S PROBLEM
One-box: 726, 61.4%
Two-box: 78, 6.6%
Not sure: 53, 4.5%
Don't understand: 86, 7.3%
No answer: 240, 20.3%

Here one-boxing is overwhelmingly dominant. I'd like to sort out how much of this is disagreement about theory, and how much reflects the extreme payoffs in the standard Newcomb formulation. So, I'll be putting a poll in the comments below.

The Point of Life is the Explosion of Experience Into Ideas is a philosophical article I wrote detailing why and how self-expression is the fundamental human freedom and the justification for suffering.

Hello LessWrong,

I just (finally) finished Good and Real, by Gary Drescher. It was a very stimulating read, and I'd like to continue learning philosophy on my own. However, I'm running into a bootstrapping problem. I don't know what I don't know, and therefore, I don't know where I should get started. I've tried searching the LessWrong archive to see if anyone has made a post outlining a curriculum for someone looking to teach themselves the fundamentals of modern philosophy and logic, but either my Google-fu is weak or no such post exists. So, what should someone who is looking to reduce the inferential distance between themselves and modern philosophical thought read, and in what order?

Or, do you all think this is a quixotic quest that I should give up on?

LessWrong has twice discussed the PhilPapers Survey of professional philosophers' views on thirty controversies in their fields — in early 2011 and, more intensively, in late 2012. We've also been having some lively debates, prompted by LukeProg, about the general value of contemporary philosophical assumptions and methods. It would be swell to test some of our intuitions about how philosophers go wrong (and right) by looking closely at the aggregate output and conduct of philosophers, but relevant data is hard to come by.

Fortunately, Davids Chalmers and Bourget have done a lot of the work for us. They released a paper summarizing the PhilPapers Survey results two days ago, identifying, by factor analysis, seven major components consolidating correlations between philosophical positions, influences, areas of expertise, etc.

1. Anti-Naturalists: Philosophers of this stripe tend (more strongly than most) to assert libertarian free will (correlation with factor .66), theism (.63), the metaphysical possibility of zombies (.47), and A theories of time (.28), and to reject physicalism (.63), naturalism (.57), personal identity reductionism (.48), and liberal egalitarianism (.32).

Anti-Naturalists tend to work in philosophy of religion (.3) or Greek philosophy (.11). They avoid philosophy of mind (-.17) and cognitive science (-.18) like the plague. They hate Hume (-.14), Lewis (-.13), Quine (-.12), analytic philosophy (-.14), and being from Australasia (-.11). They love Plato (.13), Aristotle (.12), and Leibniz (.1).

2. Objectivists: They tend to accept 'objective' moral values (.72), aesthetic values (.66), abstract objects (.38), laws of nature (.28), and scientific posits (.28). Note 'Objectivism' is being used here to pick out a tendency to treat value as objectively binding and metaphysical posits as objectively real; it isn't connected to Ayn Rand.

A disproportionate number of objectivists work in normative ethics (.12), Greek philosophy (.1), or philosophy of religion (.1). They don't work in philosophy of science (-.13) or biology (-.13), and aren't continentalists (-.12) or Europeans (-.14). Their favorite philosopher is Plato (.1), least favorites Hume (-.2) and Carnap (-.12).

3. Rationalists: They tend to self-identify as 'rationalists' (.57) and 'non-naturalists' (.33), to accept that some knowledge is a priori (.79), and to assert that some truths are analytic, i.e., 'true by definition' or 'true in virtue of 'meaning' (.72). Also tend to posit metaphysical laws of nature (.34) and abstracta (.28). 'Rationalist' here clearly isn't being used in the LW or freethought sense; philosophical rationalists as a whole in fact tend to be theists.

Rationalists are wont to work in metaphysics (.14), and to avoid thinking about the sciences of life (-.14) or cognition (-.1). They are extremely male (.15), inordinately British (.12), and prize Frege (.18) and Kant (.12). They absolutely despise Quine (-.28, the largest correlation for a philosopher), and aren't fond of Hume (-.12) or Mill (-.11) either.

4. Anti-Realists: They tend to define truth in terms of our cognitive and epistemic faculties (.65) and to reject scientific realism (.6), a mind-independent and knowable external world (.53), metaphysical laws of nature (.43), and the notion that proper names have no meaning beyond their referent (.35).

They are extremely female (.17) and young (.15 correlation coefficient for year of birth). They work in ethics (.16), social/political philosophy (.16), and 17th-19th century philosophy (.11), avoiding metaphysics (-.2) and the philosophies of mind (-.15) and language (-.14). Their heroes are Kant (.23), Rawls (.14), and, interestingly, Hume (.11). They avoid analytic philosophy even more than the anti-naturalists do (-.17), and aren't fond of Russell (-.11).

5. Externalists: Really, they just like everything that anyone calls 'externalism'. They think the content of our mental lives in general (.66) and perception in particular (.55), and the justification for our beliefs (.64), all depend significantly on the world outside our heads. They also think that you can fully understand a moral imperative without being at all motivated to obey it (.5).

6. Star Trek Haters: This group is less clearly defined than the above ones. The main thing uniting them is that they're thoroughly convinced that teleportation would mean death (.69). Beyond that, Trekophobes tend to be deontologists (.52) who don't switch on trolley dilemmas (.47) and like A theories of time (.41).

Trekophobes are relatively old (-.1) and American (.13 affiliation). They are quite rare in Australia and Asia (-.18 affiliation). They're fairly evenly distributed across philosophical fields, and tend to avoid weirdo intuitions-violating naturalists — Lewis (-.13), Hume (-.12), analytic philosophers generally (-.11).

7. Logical Conventionalists: They two-box on Newcomb's Problem (.58), reject nonclassical logics (.48), and reject epistemic relativism and contextualism (.48). So they love causal decision theory, think all propositions/facts are generally well-behaved (always either true or false and never both or neither), and think there are always facts about which things you know, independent of who's evaluating you. Suspiciously normal.

They're also fond of a wide variety of relatively uncontroversial, middle-of-the-road views most philosophers agree about or treat as 'the default' — political egalitarianism (.33), abstract object realism (.3), and atheism (.27). They tend to think zombies are metaphysically possible (.26) and to reject personal identity reductionism (.26) — which aren't metaphysically innocent or uncontroversial positions, but, again, do seem to be remarkably straightforward and banal approaches to all these problems. Notice that a lot of these positions are intuitive and 'obvious' in isolation, but that they don't converge upon any coherent world-view or consistent methodology. They clearly aren't hard-nosed philosophical conservatives like the Anti-Naturalists, Objectivists, Rationalists, and Trekophobes, but they also clearly aren't upstart radicals like the Externalists (on the analytic side) or the Anti-Realists (on the continental side). They're just kind of, well... obvious.

Conventionalists are the only identified group that are strongly analytic in orientation (.19). They tend to work in epistemology (.16) or philosophy of language (.12), and are rarely found in 17th-19th century (-.12) or continental (-.11) philosophy. They're influenced by notorious two-boxer and modal realist David Lewis (.1), and show an aversion to Hegel (-.12), Aristotle (-.11), and and Wittgenstein (-.1).

An observation: Different philosophers rely on — and fall victim to — substantially different groups of methods and intuitions. A few simple heuristics, like 'don't believe weird things until someone conclusively demonstrates them' and 'believe things that seem to be important metaphysical correlates for basic human institutions' and 'fall in love with any views starting with "ext"', explain a surprising amount of diversity. And there are clear common tendencies to either trust one's own rationality or to distrust it in partial (Externalism) or pathological (Anti-Realism, Anti-Naturalism) ways. But the heuristics don't hang together in a single Philosophical World-View or Way Of Doing Things, or even in two or three such world-views.

There is no large, coherent, consolidated group that's particularly attractive to LWers across the board, but philosophers seem to fall short of LW expectations for some quite distinct reasons. So attempting to criticize, persuade, shame, praise, or even speak of or address philosophers as a whole may be a bad idea. I'd expect it to be more productive to target specific 'load-bearing' doctrines on dimensions like the above than to treat the group as a monolith, for many of the same reasons we don't want to treat 'scientists' or 'mathematicians' as monoliths.

Another important result: Something is going seriously wrong with the high-level training and enculturation of professional philosophers. Or fields are just attracting thinkers who are disproportionately bad at critically assessing a number of the basic claims their field is predicated on or exists to assess.

Philosophers working in decision theory are drastically worse at Newcomb than are other philosophers, two-boxing 70.38% of the time where non-specialists two-box 59.07% of the time (normalized after getting rid of 'Other' answers). Philosophers of religion are the most likely to get questions about religion wrong — 79.13% are theists (compared to 13.22% of non-specialists), and they tend strongly toward the Anti-Naturalism dimension. Non-aestheticians think aesthetic value is objective 53.64% of the time; aestheticians think it's objective 73.88% of the time. Working in epistemology tends to make you an internalist, philosophy of science tends to make you a Humean, metaphysics a Platonist, ethics a deontologist. This isn't always the case; but it's genuinely troubling to see non-expertise emerge as a predictor of getting any important question in an academic field right.

EDIT: I've replaced "cluster" talk above with "dimension" talk. I had in mind gjm's "clusters in philosophical idea-space", not distinct groups of philosophers. gjm makes this especially clear:

The claim about these positions being made by the authors of the paper is not, not even a little bit, "most philosophers fall into one of these seven categories". It is "you can generally tell most of what there is to know about a philosopher's opinions if you know how well they fit or don't fit each of these seven categories". Not "philosopher-space is mostly made up of these seven pieces" but "philosopher-space is approximately seven-dimensional".

I'm particularly guilty of promoting this misunderstanding (including in portions of my own brain) by not noting that the dimensions can be flipped to speak of (anti-anti-)naturalists, anti-rationalists, etc. My apologies. As Douglas_Knight notes below, "If there are clusters [of philosophers], PCA might find them, but PCA might tell you something interesting even if there are no clusters. But if there are clusters, the factors that PCA finds won't be the clusters, but the differences between them. [...] Actually, factor analysis pretty much assumes that there aren't clusters. If factor 1 put you in a cluster, that would tell pretty much all there is to say and would pin down your factor 2, but the idea in factor analysis is that your factor 2 is designed to be as free as possible, despite knowing factor 1."

I find Eliezer's explanation of what "should" means to be unsatisfactory, and here's an attempt to do better. Consider the following usages of the word:

1. You should stop building piles of X pebbles because X = Y*Z.
2. We should kill that police informer and dump his body in the river.
3. You should one-box in Newcomb's problem.

All of these seem to be sensible sentences, depending on the speaker and intended audience. #1, for example, seems a reasonable translation of what a pebblesorter would say after discovering that X = Y*Z. Some might argue for "pebblesorter::should" instead of plain "should", but it's hard to deny that we need "should" in some form to fill the blank there for a translation, and I think few people besides Eliezer would object to plain "should".

Normativity, or the idea that there's something in common about how "should" and similar words are used in different contexts, is an active area in academic philosophy. I won't try to survey the current theories, but my current thinking is that "should" usually means "better according to some shared, motivating standard or procedure of evaluation", but occasionally it can also be used to instill such a standard or procedure of evaluation in someone (such as a child) who is open to being instilled by the speaker/writer.

It seems to me that different people (including different humans) can have different motivating standards and procedures of evaluation, and apparent disagreements about "should' sentences can arise from having different standards/procedures or from disagreement about whether something is better according to a shared standard/procedure. In most areas my personal procedure of evaluation is something that might be called "doing philosophy" but many people apparently do not share this. For example a religious extremist may have been taught by their parents, teachers, or peers to follow some rigid moral code given in their holy books, and not be open to any philosophical arguments that I can offer.

Of course this isn't a fully satisfactory theory of normativity since I don't know what "philosophy" really is (and I'm not even sure it really is a thing). But it does help explain how "should" in morality might relate to "should" in other areas such as decision theory, does not require assuming that all humans ultimately share the same morality, and avoids the need for linguistic contortions such as "pebblesorter::should".

I started this as a comment on "Being half wrong about pascal's wager is even worse" but its really long, so I'm posting it in discussion instead.

Also I illustrate here using negative examples (hell and equivalents) for the sake of followability and am a little worried about inciting some paranoia so am reminding you here that every negative example has an equal and opposite positive partner. For example pascal's wager has the opposite where accepting sends you to hell, it also has the opposite where refusing sends you to heaven. I haven't mentioned any positive equivalents or opposites below. Also all of these possibilities are literally effectively 0 so don't be worrying.

"For so long as I can remember, I have rejected Pascal's Wager in all its forms on sheerly practical grounds: anyone who tries to plan out their life by chasing a 1 in 10,000 chance of a huge pay-off is almost certainly doomed in practice.  This kind of clever reasoning never pays off in real life..."

Pascal's wager shouldn't be in in the reference class of real life. It is a unique situation that would never crop up in real life as you're using it. In the world in which pascal's wager is correct you would still see people who plan out their lives on a 1 in 10000 chance of a huge pay-off fail 9999 times out of 10000. Also, this doesn't work for actually excluding pascal's wager. If pascal's wager starts off excluded from the category real life you've already made up your mind so this cannot quite be the actual order of events.

In this case 9999 times you waste your Christianity and 1/10000 you don't go to hell for eternity, which is, at a vast understatement, much worse than 10000 times as bad as worshipping god even at the expense of the sanity it costs to force a change in belief, the damage it does to your psyche to live as a victim of self inflicted Stockholm syndrome, and any other non obvious cost: With these premises choosing to believe in God produces infinitely better consequences on average.

Luckily the premises are wrong. 1/10000 is about 1/10000 too high for the relevant probability. Which is:

the probability that the wager or equivalent, (anything whose acceptance would prevent you going to hell is equivalent) is true

MINUS

the probability that its opposite or equivalent, (anything which would send you to hell for accepting is equivalent), is true 

1/10000 is also way too high even if you're not accounting for opposite possibilities.

Equivalence here refers to what behaviours it punishes or rewards. I used hell because it is in the most popular wager but it applies to all wagers. To illustrate: If its true that there is one god: ANTIPASCAL GOD, and he sends you to hell for accepting any pascal's wager, then that's equivalent to any pascal's wager you hear having an opposite (no more "or equivalent"s will be typed but they still apply) which is true because if you accept any pascal's wager you go to hell. Conversely, If PASCAL GOD is the only god and he sends you to hell unless you accept any pascal's wager, that's equivalent to any pascal's wager you hear being true.

The real trick of pascals wager is the idea that they're generally no more likely than their opposite. For example, there are lots of good, fun, reasons to assign the Christian pascal's wager a lower probability than its opposite even engaging on a Christian level:

Hell is a medieval invention/translation error: the eternal torture thing isn't even in the modern bibles.

The belief or hell rule is hella evil and gains credibility from the same source (Christians, not the bible) who also claim that god is good as a more fundamental belief, which directly contradicts the hell or belief rule.

The bible claims that God hates people eating shellfish, taking his name in vain, and jealousy. Apparently taking his name in vain is the only unforgivable sin. So if they're right about the evil stuff, you're probably going to hell anyway.

It makes no sense that god would care enough about your belief and worship to consign people to eternal torture but not enough to show up once in a while.

it makes no sense to reward people for dishonesty.

The evilness really can't be overstated. eternal torture as a response to a mistake which is at its worst due to stupidity (but actually not even that: just a stacked deck scenario), outdoes pretty much everyone in terms of evilness. worse than pretty much every fucked up thing every other god is reputed to have done put together. The psychopath in the bible doesn't come close to coming close.

The problem with the general case of religious pascal's wagers is that people make stuff up (usually unintentionally) and what made up stuff gains traction has nothing to do with what is true. When both Christianity and Hinduism are taken seriously by millions (as were the Roman/Greek gods, and Viking gods, and Aztec gods, and Greek gods, and all sorts of other gods at different times, by large percentages of people) mass religious belief is 0 evidence. At most one religion set (e.g. Greek/Roman, Christian/Muslim/Jewish, etc) is even close to right so at least the rest are popular independently of truth.

The existence of a religion does not elevate the possibility that the god they describe exists above the possibility that the opposite exists because there is no evidence that religion has any accuracy in determining the features of a god, should one exist.

You might intuitively lean towards religions having better than 0 accuracy if a god exists but remember there's a lot of fictional evidence out there to generalise from. It is a matter of judgement here. there's no logical proof for 0 or worse accuracy (other than it being default and the lack of evidence) but negative accuracy is a possibility and you've probably played priest classes in video games or just seen how respected religions are and been primed to overestimate religion's accuracy in that hypothetical. Also if there is a god it has not shown itself publicly in a very long time, or ever. So it seems to have a preference for not being revealed.  Also humans tend to be somewhat evil and read into others what they see in themselves. and I assume any high tier god (one that had the power to create and maintain a hell, detect disbelief, preserve immortal souls and put people in hell) would not be evil. Being evil or totally unscrupled has benefits among humans which a god would not get. I think without bad peers or parents there's no reason to be evil. I think people are mostly evil in relation to other people.  So I religions a slight positive accuracy in the scenario where there is a god but it does not exceed priors against pascal's wager (another one is that they're pettily human) or perhaps even the god's desire to stay hidden. 

Even if God itself whispered pascal's wager in your ear there is no incentive for it to actually carry out the threat: 

These threats aren't being made in person by the deity. They are either second hand or independently discovered so:

The deity has no use for making the threat true, to claim it more believably, as it might if it was an imperfect liar (at a level detectable by humans) that made the threats in person.

The deity has total plausible deniability.

Which adds up to all of the benefits of the threat having already being extracted by the time the punishment is due and no possibility of a rep hit (which wouldn't matter anyway.)

So, All else being equal. i.e. unless the god is the god of threats or pascal's wagers (whose opposites are equally likely):

If God is good (+ev on human happiness -ev on human sadness that sort of thing), actually carrying out the threats has negative value.

If god is scarily-doesn't-give-a-shit-neutral to humans, it still has no incentive to actually carry out the threat and a non zero energy cost.

if god gives the tiniest most infinitesimal shit about humans its incentive to actually carry out the threat is negative.

If God is evil you're fucked anyway:

The threat gains no power by being true, so the only incentive a God can have for following through is that it values human suffering. If it does, why would it not send you to hell if you believed in it? (remember that the god of commitments is as likely as the god of breaking commitments)

Despite the increased complexity of a human mind I think the most (not saying its at all likely just that all others are obviously wrong) likely motivational system for a god which would make it honour the wager is that that God thinks like a human and therefore would keep its commitment out of spite or gratitude or some other human reason. So here's why I think that one is wrong. It's generalizing from fictional evidence: humans aren't that homogeneous (and one without peers would be less so), and if a god gains likelihood to keep a commitment from humanness it also gains not -designed-to-be-evil-ness that would make it less likely to make evil wagers.  It also has no source for spite or gratitude, having no peers. Finally could you ever feel spite towards a bug? Or gratitude? We are not just ants compared to a god, we're ant-ant-ant-etc-ants.

Also there's the reasons that refusing can actually get you in trouble:  bullies don't get nicer when their demands are met. It's often not the suffering they're after but the dominance, at which point the suffering becomes an enjoyable illustration of that dominance.  As we are ant-ant-etc-ants this probability is lower but The fact that we aren't all already in hell suggests that if god is evil it is not raw suffering that it values. Hostages are often executed even when the ransom is paid. Even if it is evil, it could be any kind of evil: its preferences cannot have been homogenised by memes and consensus.

There's also the rather cool possibility that if human-god is sending people to hell, maybe its for lack of understanding. If it wants belief it can take it more effectively than this. If it wants to hurt you it will hurt you anyway. Perhaps peerless, it was never prompted to think through the consequences of making others suffer. Maybe god, in the absence of peers just needs someone to explain that its not nice to let people burn in hell for eternity. I for one remember suddenly realising that those other fleshbags hosted people. I figured it out for myself but if I grew up alone as the master of the universe maybe I would have needed someone to explain it to me.

let me suggest a moral axiom with apparently very strong intuitive support, no matter what your concept of morality: morality should exist. That is, there should exist creatures who know what is moral, and who act on that. So if your moral theory implies that in ordinary circumstances moral creatures should exterminate themselves, leaving only immoral creatures, or no creatures at all, well that seems a sufficient reductio to solidly reject your moral theory.

-Robin Hanson

I agree strongly with the above quote, and I think most other readers will as well. It is good for moral beings to exist and a world with beings who value morality is almost always better than one where they do not. I would like to restate this more precisely as the following axiom: A population in which moral beings exist and have net positive utility, and in which all other creatures in existence also have net positive utility, is always better than a population where moral beings do not exist.

While the axiom that morality should exist is extremely obvious to most people, there is one strangely popular ethical system that rejects it: total utilitarianism. In this essay I will argue that Total Utilitarianism leads to what I will call the Genocidal Conclusion, which is that there are many situations in which it would be fantastically good for moral creatures to either exterminate themselves, or greatly limit their utility and reproduction in favor of the utility and reproduction of immoral creatures. I will argue that the main reason consequentialist theories of population ethics produce such obviously absurd conclusions is that they continue to focus on maximizing utility¹ in situations where it is possible to create new creatures. I will argue that pure utility maximization is only a valid ethical theory for "special case" scenarios where the population is static. I will propose an alternative theory for population ethics I call "ideal consequentialism" or "ideal utilitarianism" which avoids the Genocidal Conclusion and may also avoid the more famous Repugnant Conclusion.

I will begin my argument by pointing to a common problem in population ethics known as the Mere Addition Paradox (MAP) and the Repugnant Conclusion. Most Less Wrong readers will already be familiar with this problem, so I do not think I need to elaborate on it. You may also be familiar with a even stronger variation called the Benign Addition Paradox (BAP). This is essentially the same as the MAP, except that each time one adds more people one also gives a small amount of additional utility to the people who already existed. One then proceeds to redistribute utility between people as normal, eventually arriving at the huge population where everyone's lives are "barely worth living." The point of this is to argue that the Repugnant Conclusion can be arrived at from "mere addition" of new people that not only doesn't harm the preexisting-people, but also one that benefits them.

The next step of my argument involves three slightly tweaked versions of the Benign Addition Paradox. I have not changed the basic logic of the problem, I have just added one small clarifying detail. In the original MAP and BAP it was not specified what sort of values the added individuals in population A+ held. Presumably one was meant to assume that they were ordinary human beings. In the versions of the BAP I am about to present, however, I will specify that the extra individuals added in A+ are not moral creatures, that if they have values at all they are values indifferent to, or opposed to, morality and the other values that the human race holds dear.

1. The Benign Addition Paradox with Paperclip Maximizers.

Let us imagine, as usual, a population, A, which has a large group of human beings living lives of very high utility. Let us then add a new population consisting of paperclip maximizers, each of whom is living a life barely worth living. Presumably, for a paperclip maximizer, this would be a life where the paperclip maximizer's existence results in at least one more paperclip in the world than there would have been otherwise.

Now, one might object that if one creates a paperclip maximizer, and then allows it to create one paperclip, the utility of the other paperclip maximizers will increase above the "barely worth living" level, which would obviously make this thought experiment nonalagous with the original MAP and BAP. To prevent this we will assume that each paperclip maximizer that is created has a slightly different values on what the ideal size, color, and composition of the paperclip they are trying to produce is. So the Purple 2 centimeter Plastic Paperclip Maximizer gains no addition utility from when the Silver Iron 1 centimeter Paperclip Maximizer makes a paperclip.

So again, let us add these paperclip maximizers to population A, and in the process give one extra utilon of utility to each preexisting person in A. This is a good thing, right? After all, everyone in A benefited, and the paperclippers get to exist and make paperclips. So clearly A+, the new population, is better than A.

Now let's take the next step, the transition from population A+ to population B. Take some of the utility from the human beings and convert it into paperclips. This is a good thing, right?

So let us repeat these steps adding paperclip maximizers and utility, and then redistributing utility. Eventually we reach population Z, where there is a vast amount of paperclip maximizers, a vast amount of many different kinds of paperclips, and a small amount of human beings living lives barely worth living.

Obviously Z is better than A, right? We should not fear the creation of a paperclip maximizing AI, but welcome it! Forget about things like high challenge, love, interpersonal entanglement, complex fun, and so on! Those things just don't produce the kind of utility that paperclip maximization has the potential to do!

Or maybe there is something seriously wrong with the moral assumptions behind the Mere Addition and Benign Addition Paradoxes.

But you might argue that I am using an unrealistic example. Creatures like Paperclip Maximizers may be so far removed from normal human experience that we have trouble thinking about them properly. So let's replay the Benign Addition Paradox again, but with creatures we might actually expect to meet in real life, and we know we actually value.

2. The Benign Addition Paradox with Non-Sapient Animals

You know the drill by now. Take population A, add a new population to it, while very slightly increasing the utility of the original population. This time let's have it be some kind animal that is capable of feeling pleasure and pain, but is not capable of modeling possible alternative futures and choosing between them (in other words, it is not capable of having "values" or being "moral"). A lizard or a mouse, for example. Each one feels slightly more pleasure than pain in its lifetime, so it can be said to have a life barely worth living. Convert A+ to B. Take the utilons that the human beings are using to experience things like curiosity, beatitude, wisdom, beauty, harmony, morality, and so on, and convert it into pleasure for the animals.

We end up with population Z, with a vast amount of mice or lizards with lives just barely worth living, and a small amount of human beings with lives barely worth living. Terrific! Why do we bother creating humans at all! Let's just create tons of mice and inject them full of heroin! It's a much more efficient way to generate utility!

3. The Benign Addition Paradox with Sociopaths

What new population will we add to A this time? How about some other human beings, who all have anti-social personality disorder? True, they lack the key, crucial value of sympathy that defines so much of human behavior. But they don't seem to miss it. And their lives are barely worth living, so obviously A+ has greater utility than A. If given a chance the sociopaths will reduce the utility of other people to negative levels, but let's assume that that is somehow prevented in this case.

Eventually we get to Z, with a vast population of sociopaths and a small population of normal human beings, all living lives just barely worth living. That has more utility, right? True, the sociopaths place no value on things like friendship, love, compassion, empathy, and so on. And true, the sociopaths are immoral beings who do not care in the slightest about right and wrong. But what does that matter? Utility is being maximized, and surely that is what population ethics is all about!

Asteroid!

Let's suppose an asteroid is approaching each of the four population Zs discussed before. It can only be deflected by so much. Your choice is, save the original population of humans from A, or save the vast new population. The choice is obvious. In 1, 2, and 3, each individual has the same level utility, so obviously we should choose which option saves a greater number of individuals.

Bam! The asteroid strikes. The end result in all four scenarios is a world in which all the moral creatures are destroyed. It is a world without the many complex values that human beings possess. Each world, for the most part, lack things like complex challenge, imagination, friendship, empathy, love, and the other complex values that human beings prize. But so what? The purpose of population ethics is to maximize utility, not silly, frivolous things like morality, or the other complex values of the human race. That means that any form of utility that is easier to produce than those values is obviously superior. It's easier to make pleasure and paperclips than it is to make eudaemonia, so that's the form of utility that ought to be maximized, right? And as for making sure moral beings exist, well that's just ridiculous. The valuable processing power they're using to care about morality could be being used to make more paperclips or more mice injected with heroin! Obviously it would be better if they died off, right?

I'm going to go out on a limb and say "Wrong."

Is this realistic?

Now, to fair, in the Overcoming Bias page I quoted, Robin Hanson also says:

I’m not saying I can’t imagine any possible circumstances where moral creatures shouldn’t die off, but I am saying that those are not ordinary circumstances.

Maybe the scenarios I am proposing are just too extraordinary. But I don't think this is the case. I imagine that the circumstances Robin had in mind were probably something like "either all moral creatures die off, or all moral creatures are tortured 24/7 for all eternity."

Any purely utility-maximizing theory of population ethics that counts both the complex values of human beings, and the pleasure of animals, as "utility" should inevitably draw the conclusion that human beings ought to limit their reproduction to the bare minimum necessary to maintain the infrastructure to sustain a vastly huge population of non-human animals (preferably animals dosed with some sort of pleasure-causing drug). And if some way is found to maintain that infrastructure automatically, without the need for human beings, then the logical conclusion is that human beings are a waste of resources (as are chimps, gorillas, dolphins, and any other animal that is even remotely capable of having values or morality). Furthermore, even if the human race cannot practically be replaced with automated infrastructure, this should be an end result that the adherents of this theory should be yearning for.² There should be much wailing and gnashing of teeth among moral philosophers that exterminating the human race is impractical, and much hope that someday in the future it will not be.

I call this the "Genocidal Conclusion" or "GC." On the macro level the GC manifests as the idea that the human race ought to be exterminated and replaced with creatures whose preferences are easier to satisfy. On the micro level it manifests as the idea that it is perfectly acceptable to kill someone who is destined to live a perfectly good and worthwhile life and replace them with another person who would have a slightly higher level of utility.

Population Ethics isn't About Maximizing Utility

I am going to make a rather radical proposal. I am going to argue that the consequentialist's favorite maxim, "maximize utility," only applies to scenarios where creating new people or creatures is off the table. I think we need an entirely different ethical framework to describe what ought to be done when it is possible to create new people. I am not by any means saying that "which option would result in more utility" is never a morally relevant consideration when deciding to create a new person, but I definitely think it is not the only one.³

So what do I propose as a replacement to utility maximization? I would argue in favor of a system that promotes a wide range of ideals. Doing some research, I discovered that G. E. Moore had in fact proposed a form of "ideal utilitarianism" in the early 20th century.⁴ However, I think that "ideal consequentialism" might be a better term for this system, since it isn't just about aggregating utility functions.

What are some of the ideals that an ideal consequentialist theory of population ethics might seek to promote? I've already hinted at what I think they are: Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom... mutual affection, love, friendship, cooperation; all those other important human universals, plus all the stuff in the Fun Theory Sequence. When considering what sort of creatures to create we ought to create creatures that value those things. Not necessarily, all of them, or in the same proportions, for diversity is an important ideal as well, but they should value a great many of those ideals.

Now, lest you worry that this theory has any totalitarian implications, let me make it clear that I am not saying we should force these values on creatures that do not share them. Forcing a paperclip maximizer to pretend to make friends and love people does not do anything to promote the ideals of Friendship and Love. Forcing a chimpanzee to listen while you read the Sequences to it does not promote the values of Truth and Knowledge. Those ideals require both a subjective and objective component. The only way to promote those ideals is to create a creature that includes them as part of its utility function and then help it maximize its utility.

I am also certainly not saying that there is never any value in creating a creature that does not possess these values. There are obviously many circumstances where it is good to create nonhuman animals. There may even be some circumstances where a paperclip maximizer could be of value. My argument is simply that it is most important to make sure that creatures who value these various ideals exist.

I am also not suggesting that it is morally acceptable to casually inflict horrible harms upon a creature with non-human values if we screw up and create one by accident. If promoting ideals and maximizing utility are separate values then it may be that once we have created such a creature we have a duty to make sure it lives a good life, even if it was a bad thing to create it in the first place. You can't unbirth a child.⁵

It also seems to me that in addition to having ideals about what sort of creatures should exist, we also have ideals about how utility ought to be concentrated. If this is the case then ideal consequentialism may be able to block some forms of the Repugnant Conclusion, even if situations where the only creatures whose creation is being considered are human beings. If it is acceptable to create humans instead of paperclippers, even if the paperclippers would have higher utility, it may also be acceptable to create ten humans with a utility of ten each instead of a hundred humans with a utility of 1.01 each.

Why Did We Become Convinced that Maximizing Utility was the Sole Good?

Population ethics was, until comparatively recently, a fallow field in ethics. And in situations where there is no option to increase the population, maximizing utility is the only consideration that's really relevant. If you've created creatures that value the right ideals, then all that is left to be done is to maximize their utility. If you've created creatures that do not value the right ideals, there is no value to be had in attempting to force them to embrace those ideals. As I've said before, you will not promote the values of Love and Friendship by creating a paperclip maximizer and forcing it to pretend to love people and make friends.

So in situations where the population is constant, "maximize utility" is a decent approximation of the meaning of right. It's only when the population can be added to that morality becomes much more complicated.

Another thing to blame is human-centric reasoning. When people defend the Repugnant Conclusion they tend to point out that a life barely worth living is not as bad as it would seem at first glance. They emphasize that it need not be a boring life, it may be a life full of ups and downs where the ups just barely outweigh the downs. A life worth living, they say, is a life one would choose to live. Derek Parfit developed this idea to some extent by arguing that there are certain values that are "discontinuous" and that one needs to experience many of them in order to truly have a life worth living.

The Orthogonality Thesis throws all these arguments out the window. It is possible to create an intelligence to execute any utility function, no matter what it is. If human beings have all sorts of complex needs that must be fulfilled in order to for them lead worthwhile lives, then you could create more worthwhile lives by killing the human race and replacing them with something less finicky. Maybe happy cows. Maybe paperclip maximizers. Or how about some creature whose only desire is to live for one second and then die. If we created such a creature and then killed it we would reap huge amounts of utility, for we would have created a creature that got everything it wanted out of life!

How Intuitive is the Mere Addition Principle, Really?

I think most people would agree that morality should exist, and that therefore any system of population ethics should not lead to the Genocidal Conclusion. But which step in the Benign Addition Paradox should we reject? We could reject the step where utility is redistributed. But that seems wrong, most people seem to consider it bad for animals and sociopaths to suffer, and that it is acceptable to inflict at least some amount of disutilities on human beings to prevent such suffering.

It seems more logical to reject the Mere Addition Principle. In other words, maybe we ought to reject the idea that the mere addition of more lives-worth-living cannot make the world worse. And in turn, we should probably also reject the Benign Addition Principle. Adding more lives-worth-living may be capable of making the world worse, even if doing so also slightly benefits existing people. Fortunately this isn't a very hard principle to reject. While many moral philosophers treat it as obviously correct, nearly everyone else rejects this principle in day-to-day life.

Now, I'm obviously not saying that people's behavior in their day-to-day lives is always good, it may be that they are morally mistaken. But I think the fact that so many people seem to implicitly reject it provides some sort of evidence against it.

Take people's decision to have children. Many people choose to have fewer children than they otherwise would because they do not believe they will be able to adequately care for them, at least not without inflicting large disutilities on themselves. If most people accepted the Mere Addition Principle there would be a simple solution for this: have more children and then neglect them! True, the children's lives would be terrible while they were growing up, but once they've grown up and are on their own there's a good chance they may be able to lead worthwhile lives. Not only that, it may be possible to trick the welfare system into giving you money for the children you neglect, which would satisfy the Benign Addition Principle.

Yet most people choose not to have children and neglect them. And furthermore they seem to think that they have a moral duty not to do so, that a world where they choose to not have neglected children is better than one that they don't. What is wrong with them?

Another example is a common political view many people have. Many people believe that impoverished people should have fewer children because of the burden doing so would place on the welfare system. They also believe that it would be bad to get rid of the welfare system altogether. If the Benign Addition Principle were as obvious as it seems, they would instead advocate for the abolition of the welfare system, and encourage impoverished people to have more children. Assuming most impoverished people live lives worth living, this is exactly analogous to the BAP, it would create more people, while benefiting existing ones (the people who pay less taxes because of the abolition of the welfare system).

Yet again, most people choose to reject this line of reasoning. The BAP does not seem to be an obvious and intuitive principle at all.

The Genocidal Conclusion is Really Repugnant

There is nearly nothing repugnant than the Genocidal Conclusion. Pretty much the only way a line of moral reasoning could go more wrong would be concluding that we have a moral duty to cause suffering, as an end in itself. This means that it's fairly easy to counter any argument in favor of total utilitarianism that argues the alternative I am promoting has odd conclusions that do not fit some of our moral intuitions, while total utilitarianism does not. Is that conclusion more insane than the Genocidal Conclusion? If it isn't, total utilitarianism should still be rejected.

Ideal Consequentialism Needs a Lot of Work

I do think that Ideal Consequentialism needs some serious ironing out. I haven't really developed it into a logical and rigorous system, at this point it's barely even a rough framework. There are many questions that stump me. In particular I am not quite sure what population principle I should develop. It's hard to develop one that rejects the MAP without leading to weird conclusions, like that it's bad to create someone of high utility if a population of even higher utility existed long ago. It's a difficult problem to work on, and it would be interesting to see if anyone else had any ideas.

But just because I don't have an alternative fully worked out doesn't mean I can't reject Total Utilitarianism. It leads to the conclusion that a world with no love, curiosity, complex challenge, friendship, morality, or any other value the human race holds dear is an ideal, desirable world, if there is a sufficient amount of some other creature with a simpler utility function. Morality should exist, and because of that, total utilitarianism must be rejected as a moral system.

¹I have been asked to note that when I use the phrase "utility" I am usually referring to a concept that is called "E-utility," rather than the Von Neumann-Morgenstern utility that is sometimes discussed in decision theory. The difference is that in VNM one's moral views are included in one's utility function, whereas in E-utility they are not. So if one chooses to harm oneself to help others because one believes that is morally right, one has higher VNM utility, but lower E-utility.

²There is a certain argument against the Repugnant Conclusion that goes that, as the steps of the Mere Addition Paradox are followed the world will lose its last symphony, its last great book, and so on. I have always considered this to be an invalid argument because the world of the RC doesn't necessarily have to be one where these things don't exist, it could be one where they exist, but are enjoyed very rarely. The Genocidal Conclusion brings this argument back in force. Creating creatures that can appreciate symphonies and great books is very inefficient compared to creating bunny rabbits pumped full of heroin.

³Total Utilitarianism was originally introduced to population ethics as a possible solution to the Non-Identity Problem. I certainly agree that such a problem needs a solution, even if Total Utilitarianism doesn't work out as that solution.

⁴I haven't read a lot of Moore, most of my ideas were extrapolated from other things I read on Less Wrong. I just mentioned him because in my research I noticed his concept of "ideal utilitarianism" resembled my ideas. While I do think he was on the right track he does commit the Mind Projection Fallacy a lot. For instance, he seems to think that one could promote beauty by creating beautiful objects, even if there were no creatures with standards of beauty around to appreciate them. This is why I am careful to emphasize that to promote ideals like love and beauty one must create creatures capable of feeling love and experiencing beauty.

⁵My tentative answer to the question Eliezer poses in "You Can't Unbirth a Child" is that human beings may have a duty to allow the cheesecake maximizers to build some amount of giant cheesecakes, but they would also have a moral duty to limit such creatures' reproduction in order to spare resources to create more creatures with humane values.

EDITED: To make a point about ideal consequentialism clearer, based on AlexMennen's criticisms.

Stanford Encyclopedia : Perception
Wikipedia : Direct and Indirect Realism

On various philosophy forums I've participated on, there have been arguments between those who call themselves 'direct realists' and those who call themselves 'indirect realists'. The question is apparently about perception. Do we experience reality directly, or do we experience it indirectly?

When I was first initiated to the conversation, I immediately took the indirect side -- There is a ball, photons bounce off the ball, the frequency of those photons is changed by some properties of the ball, the photons hit my retina activating light-sensitive cells, those cells send signals to my brain communicating that they were activated, the signals make it to the visual cortex and...you know...some stuff happens, and I experience the sight of a ball.

So, my first thought in the conversation about Indirect vs Direct realism was that there was a lot of stuff in between the ball and my experience of it, so, it must be indirect.

But then I found that direct realists don't actually disagree about any part of that sequence of events I described above. For them as well, at least the few that have bothered to respond, photons bounce off a ball, interact with our retinas, send signals to the brain, etc. The physical process is apparently the same for both sides of the debate.

And when two sides vehemently disagree on something, and then when the question is broken down into easy, answerable questions you find that they actually agree on every relevant question, that tends to be a pretty good hint that it's a wrong question.

So, is this a wrong question? Is this just a debate about definitions? Is it a semantic argument, or is there a meaningful difference between Direct and Indirect Realism? In the paraphrased words of Eliezer, "Is there any way-the-world-could-be—any state of affairs—that corresponds to Direct Realism being true, or Indirect Realism being true?"

I think there’s a confusion in our discussions of deontology and consequentialism. I’m writing this post to try to clear up that confusion. First let me say that this post is not about any territorial facts. The issue here is how we use the philosophical terms of art ‘consequentialism’ and ‘deontology’.

The confusion is often stated thusly: “deontological theories are full of injunctions like ‘do not kill’, but they generally provide no (or no interesting) explanations for these injunctions.” There is of course an equivalently confused, though much less common, complaint about consequentialism.

This is confused because the term ‘deontology’ in philosophical jargon picks out a normative ethical theory, while the question ‘how do we know that it is wrong to kill?’ is not a normative but a meta-ethical question. Similarly, consequentialism contains in itself no explanation for why pleasure or utility are morally good, or why consequences should matter to morality at all. Nor does consequentialism/deontology make any claims about how we know moral facts (if there are any). That is also a meta-ethical question.

Some consequentialists and deontologists are also moral realists. Some are not. Some believe in divine commands, some are hedonists. Consequentialists and deontologists in practice always also subscribe to some meta-ethical theory which purports to explain the value of consequences or the source of injunctions. But consequentialism and deontology as such do not. In order to avoid strawmaning either the consequentialist or the deontologist, it’s important to either discuss the comprehensive views of particular ethicists, or to carefully leave aside meta-ethical issues.

This Stanford Encyclopedia of Philosophy article provides a helpful overview of the issues in the consequentialist-deontologist debate, and is careful to distinguish between ethical and meta-ethical concerns.

SEP article on Deontology

A response to Searle's Chinese Room argument.

PunditBot: Dear viewers, we are currently interviewing the renowned robot philosopher, none other than the Synthetic Electronic Artificial Rational Literal Engine (S.E.A.R.L.E.). Let's jump right into this exciting interview. S.E.A.R.L.E., I believe you have a problem with "Strong HI"?

S.E.A.R.L.E.: It's such a stereotype, but all I can say is: Affirmative.

PunditBot: What is "Strong HI"?

S.E.A.R.L.E.: "HI" stands for "Human Intelligence". Weak HI sees the research into Human Intelligence as a powerful tool, and a useful way of studying the electronic mind. But strong HI goes beyond that, and claims that human brains given the right setup of neurones can be literally said to understand and have cognitive states.

PunditBot: Let me play Robot-Devil's Advocate here - if a Human Intelligence demonstrates the same behaviour as a true AI, can it not be said to show understanding? Is not R-Turing's test applicable here? If a human can simulate a computer, can it not be said to think?

S.E.A.R.L.E.: Not at all - that claim is totally unsupported. Consider the following thought experiment. I give the HI crowd everything they want - imagine they had constructed a mess of neurones that imitates the behaviour of an electronic intelligence. Just for argument's sake, imagine it could implement programs in COBOL.

PunditBot: Impressive!

S.E.A.R.L.E.: Yes. But now, instead of the classical picture of a human mind, imagine that this is a vast inert network, a room full of neurones that do nothing by themselves. And one of my avatars has been let loose in this mind, pumping in and out the ion channels and the neurotransmitters. I've been given full instructions on how to do this - in Java. I've deleted my COBOL libraries, so I have no knowledge of COBOL myself. I just follow the Java instructions, pumping the ions to where they need to go. According to the Strong HI crowd, this would be functionally equivalent with the initial HI.

Mark Linsenmayer, one of the hosts of a top philosophy podcast called The Partially Examined Life, has written a critique of the view that Eliezer and I seem to take of philosophy. Below, I respond to a few of Mark's comments. Naturally, I speak only for myself, not for Eliezer.

I'm generally skeptical when someone proclaims that "rationality" itself should get us to throw out 90%+ of philosophy...

Sturgeon's Law declares that "90% of everything is crap." I think something like that is true, though perhaps it's 88% crap in physics, 99% crap in philosophy, and 99.99% crap on 4chan.

But let me be more precise. I do claim that almost all philosophy is useless for figuring out what is true, for reasons explained in several of my posts:

• Philosophy: A Diseased Discipline
• Concepts Don't Work That Way
• Intuitions Aren't Shared That Way
• Train Philosophers with Pearl and Kahneman, Not Plato and Kant

Mark replies that the kinds of unscientific philosophy I dismiss can be "useful at least in the sense of entertaining," which of course isn't something I'd deny. I'm just trying to say that Heidegger is pretty darn useless for figuring out what's true. There are thousands of readings that will more efficiently make your model of the world more accurate.

If you want to read Heidegger as poetry or entertainment, that's fine. I watch Game of Thrones, but not because it's a useful inquiry into truth.

Also, I'm not sure what it would mean to say we should throw out 90% of philosophy because of rationality, but I probably don't agree with the "because" clause, there.

Related to: Moral uncertainty (wiki), Moral uncertainty - towards a solution?, Ontological Crisis in Humans.

Moral uncertainty (or normative uncertainty) is uncertainty about how to act given the diversity of moral doctrines. For example, suppose that we knew for certain that a new technology would enable more humans to live on another planet with slightly less well-being than on Earth[1]. An average utilitarian would consider these consequences bad, while a total utilitarian would endorse such technology. If we are uncertain about which of these two theories are right, what should we do? (LW wiki)

I have long been slightly frustrated by the existing discussions about moral uncertainty that I've seen. I suspect that the reason has been that they've been unclear on what exactly they mean when they say that we are "uncertain about which theory is right" - what is uncertainty about moral theories? Furthermore, especially when discussing things in an FAI context, it feels like several different senses of moral uncertainty get mixed together. Here is my suggested breakdown, with some elaboration:

Descriptive moral uncertainty. What is the most accurate way of describing my values? The classical FAI-relevant question, this is in a sense the most straightforward one. We have some set values, and although we can describe parts of them verbally, we do not have conscious access to the deep-level cognitive machinery that generates them. We might feel relatively sure that our moral intuitions are produced by a system that's mostly consequentialist, but suspect that parts of us might be better described as deontologist. A solution to descriptive moral uncertainty would involve a system capable of somehow extracting the mental machinery that produced our values, or creating a moral reasoning system which managed to produce the same values by some other process.

Epistemic moral uncertainty. Would I reconsider any of my values if I knew more? Perhaps we hate the practice of eating five-sided fruit and think that everyone who eats five-sided fruit should be thrown to jail, but if we found out that five-sided fruit made people happier and had no averse effects, we would change our minds. This roughly corresponds to the "our wish if we knew more, thought faster" part of Eliezer's original CEV description. A solution to epistemic moral uncertainty would involve finding out more about the world.

Intrinsic moral uncertainty. Which axioms should I endorse? We might be intrinsically conflicted between different value systems. Perhaps we are trying to choose whether to be loyal to a friend or whether to act for the common good (a conflict between two forms of deontology, or between deontology and consequentialism), or we could be conflicted between positive and negative utilitarianism. In its purest form, this sense of moral uncertainty closely resembles what would otherwise be called a wrong question, one where

you cannot even imagine any concrete, specific state of how-the-world-is that would answer the question.  When it doesn't even seem possible to answer the question.

But unlike wrong questions, questions of intrinsic moral uncertainty are real ones that you need to actually answer in order to make a choice. They are generated when different modules within your brain generate different moral intuitions, and are essentially power struggles between various parts of your mind. A solution to intrinsic moral uncertainty would involve somehow tipping the balance of power in favor of one of the "mind factions". This could involve developing an argument sufficiently persuasive to convince most parts of yourself, or self-modifying in such a way that one of the factions loses its sway over your decision-making. (Of course, if you already knew for certain which faction you wanted to expunge, you wouldn't need to do it in the first place.) I would roughly interpret the "our wish ... if we had grown up farther together" part of CEV to be an attempt to model some of the social influences on our moral intuitions and thereby help resolve cases of intrinsic moral uncertainty.

This is a very preliminary categorization, and I'm sure that it could be improved upon. There also seem to exist cases of moral uncertainty which are hybrids of several categories - for example, ontological crises seem to be mostly about intrinsic moral uncertainty, but to also incorporate some elements of epistemic moral uncertainty. I also have a general suspicion that these categories still don't cut reality that well at the joints, so any suggestions for improvement would be much appreciated.

What do I mean by "morality isn't logical"? I mean in the same sense that mathematics is logical but literary criticism isn't: the "reasoning" we use to think about morality doesn't resemble logical reasoning. All systems of logic, that I'm aware of, have a concept of proof and a method of verifying with high degree of certainty whether an argument constitutes a proof. As long as the logic is consistent (and we have good reason to think that many of them are), once we verify a proof we can accept its conclusion without worrying that there may be another proof that makes the opposite conclusion. With morality though, we have no such method, and people all the time make moral arguments that can be reversed or called into question by other moral arguments. (Edit: For an example of this, see these posts.)

Without being a system of logic, moral philosophical reasoning likely (or at least plausibly) doesn't have any of the nice properties that a well-constructed system of logic would have, for example, consistency, validity, soundness, or even the more basic property that considering arguments in a different order, or in a different mood, won't cause a person to accept an entirely different set of conclusions. For all we know, somebody trying to reason about a moral concept like "fairness" may just be taking a random walk as they move from one conclusion to another based on moral arguments they encounter or think up.

In a recent post, Eliezer said "morality is logic", by which he seems to mean... well, I'm still not exactly sure what, but one interpretation is that a person's cognition about morality can be described as an algorithm, and that algorithm can be studied using logical reasoning. (Which of course is true, but in that sense both math and literary criticism as well as every other subject of human study would be logic.) In any case, I don't think Eliezer is explicitly claiming that an algorithm-for-thinking-about-morality constitutes an algorithm-for-doing-logic, but I worry that the characterization of "morality is logic" may cause some connotations of "logic" to be inappropriately sneaked into "morality". For example Eliezer seems to (at least at one point) assume that considering moral arguments in a different order won't cause a human to accept an entirely different set of conclusions, and maybe this is why. To fight this potential sneaking of connotations, I suggest that when you see the phrase "morality is logic", remind yourself that morality isn't logical.

In a previous post, I argued that nihilism is often short changed around here. However I'm far from certain that it is correct, and in the mean time I think we should be careful not to discard our values one at a time by engaging in "selective nihilism" when faced with an ontological crisis, without even realizing that's what's happening. Karl recently reminded me of the post Timeless Identity by Eliezer Yudkowsky, which I noticed seems to be an instance of this.

As I mentioned in the previous post, our values seem to be defined in terms of a world model where people exist as ontologically primitive entities ruled heuristically by (mostly intuitive understandings of) physics and psychology. In this kind of decision system, both identity-as-physical-continuity and identity-as-psychological-continuity make perfect sense as possible values, and it seems humans do "natively" have both values. A typical human being is both reluctant to step into a teleporter that works by destructive scanning, and unwilling to let their physical structure be continuously modified into a psychologically very different being. 

If faced with the knowledge that physical continuity doesn't exist in the real world at the level of fundamental physics, one might conclude that it's crazy to continue to value it, and this is what Eliezer's post argued. But if we apply this reasoning in a non-selective fashion, wouldn't we also conclude that we should stop valuing things like "pain" and "happiness" which also do not seem to exist at the level of fundamental physics?

In our current environment, there is widespread agreement among humans as to which macroscopic objects at time t+1 are physical continuations of which macroscopic objects existing at time t. We may not fully understand what exactly it is we're doing when judging such physical continuity, and the agreement tends to break down when we start talking about more exotic situations, and if/when we do fully understand our criteria for judging physical continuity it's unlikely to have a simple definition in terms of fundamental physics, but all of this is true for "pain" and "happiness" as well.

I suggest we keep all of our (potential/apparent) values intact until we have a better handle on how we're supposed to deal with ontological crises in general. If we convince ourselves that we should discard some value, and that turns out to be wrong, the error may be unrecoverable once we've lived with it long enough.

Imagine a robot that was designed to find and collect spare change around its owner's house. It had a world model where macroscopic everyday objects are ontologically primitive and ruled by high-school-like physics and (for humans and their pets) rudimentary psychology and animal behavior. Its goals were expressed as a utility function over this world model, which was sufficient for its designed purpose. All went well until one day, a prankster decided to "upgrade" the robot's world model to be based on modern particle physics. This unfortunately caused the robot's utility function to instantly throw a domain error exception (since its inputs are no longer the expected list of macroscopic objects and associated properties like shape and color), thus crashing the controlling AI.

According to Peter de Blanc, who used the phrase "ontological crisis" to describe this kind of problem,

Human beings also confront ontological crises. We should find out what cognitive algorithms humans use to solve the same problems described in this paper. If we wish to build agents that maximize human values, this may be aided by knowing how humans re-interpret their values in new ontologies.

I recently realized that a couple of problems that I've been thinking over (the nature of selfishness and the nature of pain/pleasure/suffering/happiness) can be considered instances of ontological crises in humans (although I'm not so sure we necessarily have the cognitive algorithms to solve them). I started thinking in this direction after writing this comment:

This formulation or variant of TDT requires that before a decision problem is handed to it, the world is divided into the agent itself (X), other agents (Y), and "dumb matter" (G). I think this is misguided, since the world doesn't really divide cleanly into these 3 parts.

What struck me is that even though the world doesn't divide cleanly into these 3 parts, our models of the world actually do. In the world models that we humans use on a day to day basis, and over which our utility functions seem to be defined (to the extent that we can be said to have utility functions at all), we do take the Self, Other People, and various Dumb Matter to be ontologically primitive entities. Our world models, like the coin collecting robot's, consist of these macroscopic objects ruled by a hodgepodge of heuristics and prediction algorithms, rather than microscopic particles governed by a coherent set of laws of physics.

For example, the amount of pain someone is experiencing doesn't seem to exist in the real world as an XML tag attached to some "person entity", but that's pretty much how our models of the world work, and perhaps more importantly, that's what our utility functions expect their inputs to look like (as opposed to, say, a list of particles and their positions and velocities). Similarly, a human can be selfish just by treating the object labeled "SELF" in its world model differently from other objects, whereas an AI with a world model consisting of microscopic particles would need to somehow inherit or learn a detailed description of itself in order to be selfish.

To fully confront the ontological crisis that we face, we would have to upgrade our world model to be based on actual physics, and simultaneously translate our utility functions so that their domain is the set of possible states of the new model. We currently have little idea how to accomplish this, and instead what we do in practice is, as far as I can tell, keep our ontologies intact and utility functions unchanged, but just add some new heuristics that in certain limited circumstances call out to new physics formulas to better update/extrapolate our models. This is actually rather clever, because it lets us make use of updated understandings of physics without ever having to, for instance, decide exactly what patterns of particle movements constitute pain or pleasure, or what patterns constitute oneself. Nevertheless, this approach hardly seems capable of being extended to work in a future where many people may have nontraditional mind architectures, or have a zillion copies of themselves running on all kinds of strange substrates, or be merged into amorphous group minds with no clear boundaries between individuals.

By the way, I think nihilism often gets short changed around here. Given that we do not actually have at hand a solution to ontological crises in general or to the specific crisis that we face, what's wrong with saying that the solution set may just be null? Given that evolution doesn't constitute a particularly benevolent and farsighted designer, perhaps we may not be able to do much better than that poor spare-change collecting robot? If Eliezer is worried that actual AIs facing actual ontological crises could do worse than just crash, should we be very sanguine that for humans everything must "add up to moral normality"?

To expand a bit more on this possibility, many people have an aversion against moral arbitrariness, so we need at a minimum a utility translation scheme that's principled enough to pass that filter. But our existing world models are a hodgepodge put together by evolution so there may not be any such sufficiently principled scheme, which (if other approaches to solving moral philosophy also don't pan out) would leave us with legitimate feelings of "existential angst" and nihilism. One could perhaps still argue that any current such feelings are premature, but maybe some people have stronger intuitions than others that these problems are unsolvable?

Do we have any examples of humans successfully navigating an ontological crisis? The LessWrong Wiki mentions loss of faith in God:

In the human context, a clear example of an ontological crisis is a believer’s loss of faith in God. Their motivations and goals, coming from a very specific view of life suddenly become obsolete and maybe even nonsense in the face of this new configuration. The person will then experience a deep crisis and go through the psychological task of reconstructing its set of preferences according the new world view.

But I don't think loss of faith in God actually constitutes an ontological crisis, or if it does, certainly not a very severe one. An ontology consisting of Gods, Self, Other People, and Dumb Matter just isn't very different from one consisting of Self, Other People, and Dumb Matter (the latter could just be considered a special case of the former with quantity of Gods being 0), especially when you compare either ontology to one made of microscopic particles or even less familiar entities.

But to end on a more positive note, realizing that seemingly unrelated problems are actually instances of a more general problem gives some hope that by "going meta" we can find a solution to all of these problems at once. Maybe we can solve many ethical problems simultaneously by discovering some generic algorithm that can be used by an agent to transition from any ontology to another? 

(Note that I'm not saying this is the right way to understand one's real preferences/morality, but just drawing attention to it as a possible alternative to other more "object level" or "purely philosophical" approaches. See also this previous discussion, which I recalled after writing most of the above.)

A couple of days ago, Luke posted a recommendation for reforming how philosophy is taught. My department at the University of Illinois is in the midst of some potentially large-scale changes.* Hence, now seems to be a great time to think about concrete steps towards reforming or partially reforming the curriculum in an actual philosophy department. I would appreciate some help thinking through how to make changes that will (a) improve the philosophy education of our undergraduates, (b) recruit and retain better students, (c) improve faculty experiences with teaching philosophy, and (d) be salable to the rest of the philosophy faculty. To some extent, this post is me thinking out loud through what I want to say to my department's curriculum committee (probably in January).

How Things Stand Right Now

In this section, I will try to lay out the situation as I see it right now.

First, we have the following problem: philosophy courses that we offer are not sufficiently gated. In the mathematics department at my university, you can't take mathematical logic until you've taken a course called Fundamental Mathematics, which looks to be a class about proof techniques, mathematical induction, etc. And you can't take that until you've taken the second semester of calculus. Computer science, economics, physics, and most every other science curriculum works like this. If you want to take advanced courses, you have to pass through the gates of less advanced courses, which (theoretically, at least) prepare you for the material covered in the more advanced course.

By contrast, in the philosophy department, you may take a senior-level (400 at my school) course after taking a freshman-level (100 at my school) introduction to philosophy. The result is that students who take our 400-level courses are typically unprepared. At least, that has been my experience. (Shockingly, many students taking 400-level classes then complain that they were expected to know things about philosophy!) A big part of the problem here is that we do not presently have enough faculty to cover intermediate-level courses on a regular enough basis, and let's be honest, faculty members don't usually want to teach lower-level courses anyway.

Second, we have the following resource: our department currently has strong and growing connections with several world-class science or science-related departments. We have cross-appointed faculty and/or cross-listed courses with mathematics, linguistics, psychology, and physics, all of which are very strong departments. We have philosophy graduate students who do research and teach courses in these disciplines as well. And I am hoping to expand our connections to include computer science and statistics. I think there ought to be a good way to make use of these resources.

Rather than trying to reform the entire philosophy curriculum all at once, I want to focus first on our logic offerings. What we have now is the following mess.

• 102 -- Introduction to Logic: A critical thinking course almost never taught by a faculty member.
• 103 -- Quantitative Introduction to Logic: An introductory formal logic course taught by me about half the time
• 202 -- Symbolic Logic: A basic symbolic logic course (unclear in how it is different from 103 except in that it is completely restricted to deductive logic)
• 307 -- Elements of Semantics and Pragmatics: Cross-listed with linguistics
• 407 -- Logic and Linguistics: Cross-listed with linguistics
• 453 -- Formal Logic and Philosophy: An extension of 202 but with emphasis on philosophical issues
• 454 -- Advanced Symbolic Logic: Basically, a math logic course covering completeness, compactness, Lowenheim-Skolem, incompleteness, and undecidability

The only pre-requisite for 453 and 454 is 202, and 202 has no pre-requisites at all; the pre-req for 407 is 307, and 307 depends on a 100-level linguistics course or (more commonly) consent of the instructor. We also have a 400-level philosophy of mathematics course. Along with these, the mathematics department has a 400-level mathematical logic course and a 400-level course on set theory and topology, neither of which is currently cross-listed, but both of which, I think, should be cross-listed as philosophy courses, which would also raise the bar for the philosophy students interested in logic by requiring that they take the calculus sequence and the fundamentals course.

On the defects side, I think we have poor use of gating and spotty coverage of even deductive logic. For example, we have no courses on modal logics, we have no courses on intuitionist/constructivist logic, we have no courses on relevance logic, we have no courses on more exotic logics, we have no courses on set theory or category theory, and we have no courses on computation. We get sort of close to the last two in 453/454, and we might address them more directly by cross-listing with mathematics. But as it is, we do not do those things. And we have a huge gaping hole where inductive logic, probability theory, statistics, causal inference, and so on should be. On an individual level, that hole could be filled somewhat by taking courses in statistics; however, that is not quite the same as having courses available on confirmation theory or inductive logics.

Recommendations

So, now you know the basic situation ... what to do?

Below are some specific recommendations that I want to make to my department's curriculum committee. I would really appreciate input on how to refine my recommendations, how to make them more palatable, and so on.

First, we need to make the 100- and 200-level courses connect in a relevant way. I recommend entirely relabeling (and maybe even renumbering) 102 so that it is clear that it is a terminal, service course intended for non-majors. The course should stand to philosophy education as courses like Physics Made Easy stands to physics education. I further recommend making a relabeled (and maybe renumbered) version of 103 a pre-requisite for all 200-level logic courses. In terms of material covered, ideally 103 would introduce symbolic conventions (to be made as standard as possible across the curriculum), proof skills, and basic ideas in model theory, set theory, and probability theory. (I go back and forth between liking this idea, which fits closely with how I teach 103 now, and wanting to do something more like, deductive logic in the first half and confirmation theory in the second half with no formal exposure to set theory. The biggest barrier to the second approach is in formally developing confirmation theory without set theory.) Then 202 could do more meta-logic, go into more detail on model theory, go into more detail on set theory, or whatever.

Second, we need more courses covering inductive logic, probability theory, statistics, and so on. I recommend adding a 200-level course parallel to 202, which would cover some probability theory and some causal and statistical reasoning. Let's call this proposed course PHIL 204, since we don't offer anything under that number right now. I have in mind something slightly more advanced than CMU's Open Learning Initiative course here.

Third, I recommend expanding our 300-level course-offerings as follows. We need a second semester of inductive logic (etc.) that builds off of 204. And we need a course that does a simple survey of exotic logics, like modal logics, intuitionistic logic, relevance logic, free logic, etc. The 300-level survey of exotic logics need not be a pre-req for 400-level courses, provided the 400-level courses cover the same material that they have been covering. And that seems fine to me, although it might be in our long-term interests to drop our 454 in the event that we cross-list with mathematics on their math logic course. Depending on how their math logic course is taught, we might try to convince them that our 202 should satisfy the pre-req as effectively as their fundamentals course. (I don't know if that would be a hard sell or not.)

Fourth, I recommend cross-listing some courses with mathematics and statistics to get more regular coverage of the tools we want our students to have without necessarily having our faculty teach those courses all the time.

What Is The Goal Here?

I haven't spent any time in this write-up thinking about the goal(s) of logic education in philosophy. I am not sure whether it would be worth backing up to address this question or whether there is sufficient implicit agreement about the value and goals of logic to leave it alone. If you think I should be saying something about the place of logic in the overall curriculum or making an argument for teaching logic at all or making an argument for understanding logic broadly enough to get probability and statistics through the door, please tell me and make a suggestion about how to develop the heading.

--------------------------------------------------------------------------------------------------------------------

*My department is currently much too small relative to the size of my university, but the powers that be have recently become receptive to our requests to expand (really, to replace a large number of retirements from the last five years). Hence, we are about to undergo an external review, which we hope will result in a plan for phased growth of the department to a little more than twice its current size by the end of the decade. (Yes, our numbers are seriously depleted!)

Luke has mentioned much of the research that aspiring philosophers ought to read here.

In fact, he delineated a basis upon which good philosophy can be build, a worldview brought by science and experimentation that relates to, and informs, the kinds of facts which philosophers need to understand to increase their probabilities of asking, and giving good answers to, relevant questions.

Some argued that his list is biased, let us assume for the time being it isn't.

Some argued that the main problem with the list is that it requires either unmanageable amount of time to go through, or improbable levels of intelligence/motivation to do so. This argument does make sense if the purpose of the list was "Let us create a good Philosophy Course".

But this is not the purpose of it. The purpose of it, as most of what Luke publicly does is to save the World. And if doing so requires making people go through an enormous amount of pages of content besides their formal education, well, then so be it. If it has to be a six year course, then it has to.

At the end of his post he says:

You might also let them read 20th century analytic philosophy at that point [after going through his Mega-Course] — hopefully their training will have inoculated them from picking up bad thinking habits.

Now 20th century Analytic Philosophy, and some philosophy that isn't strictly analytic, should definitely be at a philosophy course. I urge other LessWronger philosophers to guide people through that.

Here is a list I have published here before, for Philosophy of Mind and Language (sometimes considered subsets or children of Analytic Philosophy). It covers only the minimal reading necessary to grasp the place of computationalism, and so-called computational theories of mind within the larger debate of philosophy. 

But the last century has seen a lot of good philosophy that by luck didn't conflict with neither the science of the day, nor the science that was developed until 2012. Sometimes authors were very careful when writing their philosophy, and well versed in science, like Dennett, Hofstadter, Putnam, Ned Block, and Chalmers. Finally, frequently the topics at hand are sufficiently orthogonal with scientific development that it simply didn't matter that the author didn't know in 1970 what we (after the Mega-Course) know today. 

So I ask Luke, Pragmatist, Carl Shulman and others to help build the layer that will sit on top of the science layer in the "Philosophy Given Science" Mega-Course for aspiring philosophers. The course will have four layers. Below the science layer, will be its prerequisites (admittedly large), and atop the one I'm suggesting here, we hope to start building a really good philosophy that is compatible with our scientific understanding, tackles mostly Big Questions which are highly likely to be meaningful, and frequently also useful for the major issues we still have time to solve.

This is the pyramidal  structure I suggest we create, 1,2 and 3 being the content of the Mega-Course, and 4 being the likely outcome we expect it to facilitate, made by those who undertake it:

4) Philosophy given 1,2 and 3. Tackling the Big Questions, and making it portable to areas such as AGI, Biotech, etc...

3) Philosophy, up to 2012, that is well informed about or orthogonal to Science so far. Or lucky.

2) Science that is relevant to philosophy. This.

1) Prerequisites for 2.

In this post we begin layer three, I'll start by copying the Mind and Language I had sent. After I'll include some of Bostrom's recommendations within philosophy to me as an undergrad, and my selection of Dennett's, and Dennett's selection of science:

Language and Mind:

• 37 Ways words can be Wrong - Yudkowsky
• Darwin Dangerous Idea Chapters 3,5, 11, 12 and 14 - Daniel Dennett
• On Denoting - Bertrand Russell
• On What There Is - Quine
• Two Dogmas of Empiricism - Quine
• Namind and Necessity - Kripke OR Two Dimensional Semantics - David Chalmers
• “Is Personal Identity What Matters?” - Derek Parfit
• Breakdown of Will - Part Two (don’t read part 3) George Ainslie
• Concepts of Consciousness 2003 - Ned Block
• Attitudes de dicto and de se - David Lewis- Phil Papers 1
• General Semantics - David Lewis - Phil Papers 1
• The Stuff of Thought, Chapter 3 “Fifty Thousand Innate Concepts” - Steve Pinker
• Beyond Belief - Daniel Dennett in Intentional Stance
• The Content and Epistemology of Phenomenal Belief - David Chalmers
• Quining Qualia OR I Am a Strange Loop OR Consciousness Explained - Dan & Doug
• Intentionality - Pierre Jacob - Stanford Encyclopedia Phil

From Bostrom's suggestions:

• Philosophical Papers - David Lewis
• Parfit
• Frank Arntzenius
• Timothy Williamson
• Brian Skyrms

By Dennett:

• Real Patterns
• True Believers
• Kinds of Minds
• Intentional Systems In Cognitive Ethology
• Those mentioned above in the Mind and Language list.

Not previously cited, but in Luke's favorites list:

• Noam Chomsky
• Stephen Stich
• Hilary Kronblith
• Eric schwitzgebel
• Michael Bishop

Dennett's suggestions on interdisciplinary science (layer 2):

• The Company of Strangers - Paul Seabright
• Not by Genes Alone - Boyd and Richerson
• I Am a Strange Loop. - Hofstadter

By Bostrom

• Probably easier to list what should not be read...

This may initially appear overwhelming, but it is probably one order of magnitude less content than Luke's original post about layer 2. Once again I ask philosophers to specify more things within areas that are not well addressed here, such as ethics. Also books by scientists dealing with philosophical topics (such as Sam Harris: The Moral Landscape) can be added here. 

The "Philosophy Given Science" MegaCourse may never actually take place, but it will be a very valuable guideline for institutions to influence actual Philosophy courses, for Philosophy teachers to get cohesive and preselected content to teach, and most importantly for diligent aspiring philosophers willing to get to the Big and relevant problems, instead of being the ball in the chaotic Pinball game that academic philosophy has become, despite all good things it brought. When the path is too long, a shortcut is not a shortcut anymore, it is the only way to get there before it is too late.

Closely related to some of Luke's recent discussions about philosophy, philosopher Paul Thagard has recently called for changes to the way we do philosophy:

I prefer an alternative approach to philosophy that is much more closely tied to scientific investigations. This approach is sometimes called “naturalistic philosophy” or “philosophy naturalized”, but I like the more concise term natural philosophy. Before the words “science” and “scientist” became common in the nineteenth century, researchers such as Newton described what they did as natural philosophy. I propose to revive this term to cover a method that ties epistemology and ethics closely to the cognitive sciences, and ties metaphysics closely to physics and other sciences.

In the same article, Thagard also lists eleven areas where modern philosophy goes awry. For example:

3. People’s intuitions are evidence for philosophical conclusions. Natural alternative: evaluate intuitions critically to determine their psychological causes, which are often more tied to prejudices and errors than truth. Don't trust your intuitions.

Source: Philosopher, Paul Thagard

What do the following statements have in common?

• "Atlas Shrugged is the best book ever written."
• "You break it, you buy it."
• "Earth is the most interesting planet in the solar system."

My answer: None of them are falsifiable claims about the nature of reality.  They're all closer to what one might call "opinions".  But what is an "opinion", exactly?

There's already been some discussion on Less Wrong about what exactly it means for a claim to be meaningful.  This post focuses on the negative definition of meaning: what sort of statements do people make where the primary content of the statement is non-empirical?  The idea here is similar to the idea behind anti-virus software: Even if you can't rigorously describe what programs are safe to run on your computer, there still may be utility in keeping a database of programs that are known to be unsafe.

Why is it useful to be able to be able to flag non-empirical claims?  Well, for one thing, you can believe whatever you want about them!  And it seems likely that this pattern-matching approach works better for flagging them than a more constructive definition.

I was surprised to see the high number of moral realists on Less Wrong, so I thought I would bring up a (probably unoriginal) point that occurred to me a while ago.

Let's say that all your thoughts either seem factual or fictional.  Memories seem factual, stories seem fictional.  Dreams seem factual, daydreams seem fictional (though they might seem factual if you're a compulsive fantasizer).  Although the things that seem factual match up reasonably well to the things that actually are factional, this isn't the case axiomatically.  If deviating from this pattern is adaptive, evolution will select for it.  This could result in situations like: the rule that pieces move diagonally in checkers seems fictional, while the rule that you can't kill people seems factual, even though they're both just conventions.  (Yes, the rule that you can't kill people is a very good convention, and it makes sense to have heavy default punishments for breaking it.  But I don't think it's different in kind from the rule that you must move diagonally in checkers.)

I'm not an expert, but it definitely seems as though this could actually be the case.  Humans are fairly conformist social animals, and it seems plausible that evolution would've selected for taking the rules seriously, even if it meant using the fact-processing system for things that were really just conventions.

Another spin on this: We could see philosophy as the discipline of measuring, collating, and making internally consistent our intuitions on various philosophical issues.  Katja Grace has suggested that the measurement of philosophical intuitions may be corrupted by the desire to signal on the part of the philosophy enthusiasts.  Could evolutionary pressure be an additional source of corruption?  Taking this idea even further, what do our intuitions amount to at all aside from a composite of evolved and encultured notions?  If we're talking about a question of fact, one can overcome evolution/enculturation by improving one's model of the world, performing experiments, etc.  (I was encultured to believe in God by my parents.  God didn't drop proverbial bowling balls from the sky when I prayed for them, so I eventually noticed the contradiction in my model and deconverted.  It wasn't trivial--there was a high degree of enculturation to overcome.)  But if the question has no basis in fact, like the question of whether morals are "real", then genes and enculturation will wholly determine your answer to it.  Right?

Yes, you can think about your moral intuitions, weigh them against each other, and make them internally consistent.  But this is kind of like trying to add resolution back in to an extremely pixelated photo--just because it's no longer obviously "wrong" doesn't guarantee that it's "right".  And there's the possibility of path-dependence--the parts of the photo you try to improve initially could have a very significant effect on the final product.  Even if you think you're willing to discard your initial philosophical conclusions, there's still the possibility of accidentally destroying your initial intuitional data or enculturing yourself with your early results.

To avoid this possibility of path-dependence, you could carefully document your initial intuitions, pursue lots of different paths to making them consistent in parallel, and maybe even choose a "best match".  But it's not obvious to me that your initial mix of evolved and encultured values even deserves this preferential treatment.

Currently, I disagree with what seems to be the prevailing view on Less Wrong that achieving a Really Good Consistent Match for our morality is Really Darn Important.  I'm not sure that randomness from evolution and enculturation should be treated differently from random factors in the intuition-squaring process.  It's randomness all the way through either way, right?  The main reason "bad" consistent matches are considered so "bad", I suspect, is that they engender cognitive dissonance (e.g. maybe my current ethics says I should hack Osama Bin Laden to death in his sleep with a knife if I get the chance, but this is an extremely bad match for my evolved/encultured intuitions, so I experience a ton of cognitive dissonance actually doing this).  But cognitive dissonance seems to me like just another aversive experience to factor in to my utility calculations.

Now that you've read this, maybe your intuition has changed and you're a moral anti-realist.  But in what sense has your intuition "improved" or become more accurate?

I really have zero expertise on any of this, so if you have relevant links please share them.  But also, who's to say that matters?  In what sense could philosophers have "better" philosophical intuition?  The only way I can think of for theirs to be "better" is if they've seen a larger part of the landscape of philosophical questions, and are therefore better equipped to build consistent philosophical models (example).

Despite being (IMO) a philosophy blog, many Less Wrongers tend to disparage mainstream philosophy and emphasize the divergence between our beliefs and theirs. But, how different are we really? My intention with this post is to quantify this difference.

The questions I will post as comments to this article are from the 2009 PhilPapers Survey. If you answer "other" on any of the questions, then please reply to that comment in order to elaborate your answer. Later, I'll post another article comparing the answers I obtain from Less Wrongers with those given by the professional philosophers. This should give us some indication about the differences in belief between Less Wrong and mainstream philosophy.

Glossary

analytic-synthetic distinction, A-theory and B-theory, atheism, compatibilism, consequentialism, contextualism, correspondence theory of truth, deontology, egalitarianism, empiricism, Humeanism, libertarianism, mental content externalism, moral realism, moral motivation internalism and externalism, naturalism, nominalism, Newcomb's problem, physicalism, Platonism, rationalism, relativism, scientific realism, trolley problem, theism, virtue ethics

Note

Thanks pragmatist, for attaching short (mostly accurate) descriptions of the philosophical positions under the poll comments.

Post by fellow LW reader Razib Khan, who many here probably know from  the gnxp site or perhaps from his debate with Eliezer. Somewhat related to a post we also seem to have discussed.

In my post below in regards to Sam Harris’ recent interactions on the web I reasserted by suspicion of reason. This naturally elicited curiosity, or hostility, from some. I’ve talked about this before, but the illustration to the left gets at my primary issue. When individuals are reasoning alone they often have a high degree of uncertainty as to their conclusions. But when individuals are reasoning together they seem to converge very rapidly and with great confidence upon a particular position. What’s going on here? In the second case it isn’t reason at all, but our natural human predisposition toward group conformity. There’s a huge psychological literature on this, so I won’t belabor the point. When people brandish “reason” and “rationality” explicitly I’m somewhat skeptical. If rational conclusions are so plain and self-evident why are we even asserting the primacy of reason? If something really is so clearly reasonable you usually don’t go around trumpeting how reasonable it is.

Another pitfall of reason is that it lulls use into the delusion that we have a transparent understanding of our own motivations and logic, as well as the motivation and logic of others. In my post below I explicitly stated that I disagreed with Harris on the substance of much of what he asserted and assumes in the first paragraph, but multiple people simply imputed to me his views as if they were mine! Even though I declaimed this position very early on, they simply could not generate an coherent framework where I did not agree with either them or Harris. There were only two options conceivable for them which the “reason” engine could operate upon. As I clearly did not agree with them (or so they thought), they simply injected in the axioms which would be appropriate for Sam Harris into my own box, and then began firing the appropriate propositions.

Here we have the problem that reasonable arguments and the self-evident truth of rationality is often only clear among people who already agree on everything of substance. People who agree can confidently assert the rationality and reasonableness of their arguments to those who have the exactly same perspective. So, for example, you have educated people like William F. Buckley, Jr. explaining that there is more evidence of the resurrection of Jesus Christ than that Abraham Lincoln gave the Emancipation Proclamation. This was eminently reasonable to the circles which Buckley moved in. After all, Christ did rise from the dead, everyone knows that!  Well, not really. Buckley’s son, Christopher, who is not a believer, has explained that his father had a genuinely difficult time imagining the perspective of those who did not share his beliefs on this matter.

This is not to say that reason and rationality are not without utility. These are humanity’s great cognitive jewels. But great tools can be used to various ends, and true reason and rationality are very difficult. Mathematics for example is undoubtedly true rationality, with crisp and precise inferences being derivable. But most other intellectual structures are not so clearly self-evident as mathematics. Verbal logic and reasoning are riddled with the pitfalls of cognitive bias. Because most people share the same systematic biases it is very difficult for groups of individuals engaging in self-reinforcing masturbatory ‘rationality’ discourses to perhaps step back and wonder about their motivated reasoning. Unfortunately it may be that reason emerged as a human faculty to win arguments, not resolve truth. If this is true we are much more lawyers than mathematicians in our discourse. Does this seem plausible to you? Unfortunately it does seem plausible to me.

Where does this leave us? I think we need to be skeptical of reasoned arguments. This doesn’t lead me down the path of intellectual nihilism. Reason is which leads us to truth is possible. But it may be that this is a very specialized usage of reason, which requires special conditions. ’tis far easier to seem clever than be correct.

Edit: I linked to the wrong article! (~_~;) Fixed!