I've talked earlier about integral and differential ethics, in the context of population ethics. The idea is that the argument for the repugnant conclusion (and its associate, the very repugnant conclusion) is dependent on a series of trillions of steps, each of which are intuitively acceptable (adding happy people, making happiness more equal), but reaching a conclusion that is intuitively bad - namely, that we can improve the world by creating trillions of people in torturous and unremitting agony, as long as balance it out by creating enough happy people as well.

Differential reasoning accepts each step, and concludes that the repugnant conclusions are actually acceptable, because each step is sound. Integral reasoning accepts that the repugnant conclusion is repugnant, and concludes that some step along the way must therefore be rejected.

Notice that key word, "therefore". Some intermediate step is rejected, but not for intrinsic reasons, but purely because of the consequence. There is nothing special about the step that is rejected, it's just a relatively arbitrary barrier to stop the process (compare with the paradox of the heap).

Indeed, things can go awry when people attempt to fix the repugnant conclusion (a conclusion they rejected through integral reasoning) using differential methods. Things like the "person-affecting view" have their own ridiculousness and paradoxes (it's ok to bring a baby into the world if it will have a miserable life; we don't need to care about future generations if we randomise conceptions, etc...) and I would posit that it's because they are trying to fix global/integral issues using local/differential tools.

The relevance of this? It seems that integral tools might be better suited to deal with the bad convergence of AI problem. We could set up plausibly intuitive differential criteria (such as self-consistency), but institute overriding integral criteria that can override these if they go too far. I think there may be some interesting ideas in that area, potentially. The cost is that integral ideas are generally seen as less elegant, or harder to justify.

In population ethics...

Most people start out believing that the following are true:

1. That adding more happy lives is a net positive.
2. That redistributing happiness more fairly is not a net negative.
3. That the repugnant conclusion is indeed repugnant.

Some will baulk on the first statement on equality grounds, but most people should accept those three statements as presented. Then they find out about the mere addition paradox.

Someone who then accepts the repugnant could then reason something like this:

Adding happy people and redistributing fairly happiness, if done many, many times, in the way described above, will result in a repugnant conclusion. Each step along the way seems solid, but the conclusion seems wrong. Therefore I will accept the repugnant conclusion, not on its own merits, but because each step is clearly intuitively correct.

Call this the "differential" (or local) way or reasoning about population ethics. As long as each small change seems intuitively an improvement, then the global change must also be.

Adding happy people and redistributing fairly happiness, if done many, many times, in the way described above, will result in a repugnant conclusion. Each step along the way seems solid, but the conclusion seems wrong. Therefore I will reject (at least) one step, not on its own merits, but because the conclusion is clearly intuitively incorrect.

Call this the "integral" (or global) way of reasoning about population ethics. As long as the overall change seems intuitively a deterioration, then some of the small changes along the way must also be.

In general...

Now, I personally tend towards integral rather than differential reasoning on this particular topic. However, I want to make a more general point: philosophy may be over dedicated to differential reasoning. Mainly because it's easy: you can take things apart, simplify them, abstract details away, and appeal to simple principles - and avoid many potential biases along the way.

But it's also a very destructive tool to use in areas where concepts are unclear and cannot easily be made clear. Take the statement "human life is valuable". This can be taken apart quite easily, critiqued from all directions, its lack of easily described meaning its weakness. Nevertheless, integral reasoning is almost always applied: something called "human life" is taken to be "valuable", and many caveats and subdefinitions can be added to these terms without changing the fundamental (integral) acceptance of the statement. If we followed the differential approach, we might end up with the definition of "human life" as "energy exchange across a neurone cell membrane" or something equally ridiculous but much more rigorous.

Now, that example is a parody... but only because no-one sensible does that, we know that we'd lose too much value from that kind of definition. We want to build an extensive/integral definition of life, using our analysis to add clarity rather than simplify to a few core underlying concepts. But in population ethics and many other cases, we do feel free to use differential ethics, replacing vague overarching concepts with clear simplified versions that clearly throw away a lot of the initial concept.

Maybe we do it too much. To pick an example I disagree with (always a good habit), maybe there is such a thing as "society", for instance, not simply the total of individuals and their interactions. You can already use pretty crude consequentialist arguments with "societies" as agents subject to predictable actions and reactions (social science does it all the time), but what if we tried to build a rigorous definition of society as something morally valuable, rather than focusing on individual?

Anyway, we should be aware when, in arguments, we are keeping the broad goal and making the small steps and definitions conform to it, and when we are focusing on the small steps and definitions and following them wherever they lead.

Back in 2012 when visiting Leverage Research, I was amazed by the level of cooperation in daily situations I got from Mark. Mark wasn't just nice, or kind, or generous. Mark seemed to be playing a different game than everyone else.

If someone needed X, and Mark had X, he would provide X to them. This was true for lending, but also for giving away.

If there was a situation in which someone needed to direct attention to a particular topic, Mark would do it.

You get the picture. Faced with prisoner dilemmas, Mark would cooperate. Faced with tragedy of the commons, Mark would cooperate. Faced with non-egalitarian distributions of resources, time or luck (which are convoluted forms of the dictator game), Mark would rearrange resources without any indexical evaluation. The action would be the same, and the consequentialist one, regardless of which side of a dispute was the Mark side.

I never got over that impression. The impression that I could try to be as cooperative as my idealized fiction of Mark was.

In game theoretic terms, Mark was a Cooperational agent.

1. Altruistic - MaxOther
2. Cooperational - MaxSum
3. Individualist - MaxOwn
4. Equalitarian - MinDiff
5. Competitive - MaxDiff
6. Aggressive - MinOther

Under these definitions of kinds of agents used in research on game theoretical scenarios, what we call Effective Altruism would be called Effective Cooperation. The reason why we call it "altruism" is because even the most parochial EA's care about a set containing a minimum of 7 billion minds, where to a first approximation MaxSum ≈ MaxOther.

Locally however the distinction makes sense. In biology Altruism usually refers to a third concept, different from both the "A" in EA, and Alt, it means acting in such a way that Other>Own without reference to maximizing or minimizing, since evolution designs adaptation executors, not maximizers.

A globally Cooperational agent acts as a consequentialist globally. So does an Alt agent.

The question then is,

How should a consequentialist act locally?

The mathematical response is obviously as a Coo. What real people do is a mix of Coo and Ind.

My suggestion is that we use our undesirable yet unavoidable moral tribe distinction instinct, the one that separates Us from Them, and act always as Coos with Effective Altruists and mix Coo and Ind only with non EAs. That is what Mark did.

This is a small section on a paper I'm writing on moral enhancement. I'm trying to briefly summarize some of the points which were already made concerning local optima in evolutionary process and safety regarding taking humanity out of those local optima. You might find the text helpful in that it summarizes a very important concept. I don't think there's nothing new here, but I hope the way I tried to more properly phrase the utility-design trajectory space topology at the end can be fruitful. I would appreciate any insights you might have about that formulation in the end, how to better develop it more rigorously and some consequences. I do have some ideas, but I would want to hear what you have to say first.  Any other kind of general feedback on the text is also welcomed. But keep in mind this is just a section of a larger paper and I'm mainly interested in how to develop and what are the consequences of the framework at the end, rather than in properly developing any points in the middle.

Local optima are points where every nearby reachable positions are worse off, but there is at least one far away position which is vastly better. A strong case has been made that evolution often gets stuck on such local optima. In evolutionary processes, fitness is a monotonic function, i.e., it will necessarily increase or be maintained, any decrease in fitness will always be selected against. If there are vastly better solutions (for, e.g., solving cooperation problems) but in order to achieve those solutions organisms would have to pass through a lesser fit step, evolution will never reach that vastly better configuration. Evolutionary processes are limited by the topology of the fitness-design trajectory space, it can only go from design x to design y if there is at least one trajectory from x to y which is flat or ascendant, any trajectory momentarily descendent cannot be taken by the evolutionary processes. Say one is on the cyan ring ridge of the colored graphic. Although there is a vastly better configuration on the red peak, one would have to travel through the blue moat in order to get there. Unless one is a process who could pass through a sharp decrease in fitness, there would be no way of improving towards the red peak. Evolution is particularly prone to local optima due to fitness monotonicity. Enhancing human beings with the use of technology does not fall prey to the fitness monotonicity or any sort of utility monotonicity in general, we could initially make changes which would be harmful in order to latter achieve a vastly better configuration. Therefore, it seems plausible there would be a technological path out of evolution’s local optimum whereby we could rescue our species from these evolutionary imprisonments. Moreover, it is considered evolutionary local optima can be easily identifiable provided a careful, evolutionary and technical informed analysis is made. Hence these would be low-hanging fruits in the task for improving evolutionary products such as humans, easily accessible and able to produce great advances to humanity with little effort.

Nevertheless, it should be noted getting out of evolutionary local optima might not always be easy or even possible. Fitness does have a relatively strong correlation with overall human utility. And although human intelligence is not so dull as evolutionary process and does accept a decrease in utility in order to achieve a better design in the end, if the downward moat is deep enough, the risk of catastrophe - or much worse, extinction -, might not be worth taking. At least by being monotonic on a dimension correlated with utility, evolution was able to rightly avoid extreme losses. Perform widespread willy-nilly human enhancement, and we might fall on the moat guarding utility-design space garden’s delicious low-hanging fruits and not come back up. Particularly so in the case of moral enhancement, there is a self-reinforcing aspect of changing morality, motivations, values and desires. It might be the case tampering with deep and fundamental human morality is irreversible, because once we fundamentally value something else, we would not have any compelling reason for wanting to come back to our old values, desires or aspirations. Thus, it seems there are indeed cases where a small step past the edge of the moat will lead us to an irreversible path. To correctly map how each technology shapes utility-design trajectory space topology is a task deeply needed in order to carefully avoid falling on moats while attempting to reach local optima low-hanging fruits, or on even more dangerous existential holes. We ought to better get stuck at local optima than absolute minima. 

Utility-design trajectory space could be more properly defined as a space on R^n+u , a point there would use n-coordinates to locate all physically possible designs in all relevant dimensions n, it is defined by the laws of physics and by an utility function on u. A point will correspond to a design a iff all its neighbouring points x correspond to designs one physical step away from design a. Emergent designer processes such as evolution, human enhancement and AIs draw shapes on R^n+u by connecting points that are linked by one possible step under that process. Evolution’s hand is monotonic on dimension f, fitness, which makes for a pretty clumsy drawing. Biochemical human enhancement can more freely vary on f, but might contain other constraints elsewhere, that, e.g., uploaded minds would not. Extinctions correspond to singularities on u, once reached no other point is reachable, it designates lack of design. These points that can be reached but cannot reach need to be correctly mapped. It would also be relevant to investigate how each technology draws its specific shape on design space. Using u as some height analogue, some technologies might be inherently prone to shape moats with peaks on the middle, extinctions holes, effortless utility maximizing curves and so on. I believe moral enhancement draws a particularly bumpy hole-prone shape. FAI an ever utility-ascending shape, with all mishaps being existential holes.