I think that this post has something to say about political philosophy. The problem as I see it is that we want to understand how our local decision-making affects the global picture and what constraints should we put on our local decisions. This is extremely important because, arguably, people make a lot of local decisions that make us globally worse off: such as pollution ("externalities" in econo-speak). I don't buy the author's belief that we should ignore these global constraints: they are clearly important--indeed its the fear of the potential global outcomes of careless local decision-making that arguably led to the creation of this website.
However, just like a computers we have a lot of trouble integrating the global constraints into our decision-making (which is necessarily a local operation), and we probably have a great deal of bias in our estimates of what is the morally best set of choices for us to make. Just like the algorithm we would like to find some way to make the computational burden on us less in order to achieve these moral ends.
There is an approach in economics to understand social norms advocated by Herbert Gintis [PDF] that is able to analyze these sorts of scenarios. The essential idea is this: agents can engage in multiple correlated equilibria (these are a generalized version of Nash equilibria) possible as a result of various social norms. These correlated equilibria are, in a sense, patched together by a social norm from the "rational" (self-interested, local expected utility maximizers) agents' decisions. Human rights could definitely be understood in this light (I think: I haven't actually worked out the model).
Similar reasoning may also be used to understand certain types of laws and government policies. It is via these institutions (norms, human organizations, etc.) that we may efficiently impose global constraints on people's local decision-making. The karma system, for instance, on Less wrong probably changes the way that people make their decision to comment.
There is a probably a computer science - economics crossover paper here that would describe how institutions can lower the computational burden on individuals in their decision-making: so that when individuals make decisions in these simpler domains we can be sure that we will still be globally better off.
One word of caution is that this is precisely the rational behind "command economies" and these didn't work out so well during the 20th century. So choosing the "patching together" institution well is absolutely essential.
This is a post about moral philosophy, approached with a mathematical metaphor.
Here's an interesting problem in mathematics. Let's say you have a graph, made up of vertices and edges, with weights assigned to the edges. Think of the vertices as US cities and the edges as roads between them; the weight on each road is the length of the road. Now, knowing only this information, can you draw a map of the US on a sheet of paper? In mathematical terms, is there an isometric embedding of this graph in two-dimensional Euclidean space?
When you think about this for a minute, it's clear that this is a problem about reconciling the local and the global. Start with New York and all its neighboring cities. You have a sort of star shape. You can certainly draw this on the plane; in fact, you have many degrees of freedom; you can arbitrarily pick one way to draw it. Now start adding more cities and more roads, and eventually the degrees of freedom diminish. If you made the wrong choices earlier on, you might paint yourself in a corner and have no way to keep all the distances consistent when you add a new city. This is known as a "synchronization problem." Getting it to work locally is easy; getting all the local pieces reconciled with each other is hard.
This is a lovely problem and some acquaintances of mine have written a paper about it. (http://www.math.princeton.edu/~mcucurin/Sensors_ASAP_TOSN_final.pdf) I'll pick out some insights that seem relevant to what follows. First, some obvious approaches don't work very well. It might be thought we want to optimize over all possible embeddings, picking the one that has the lowest error in approximating distances between cities. You come up with a "penalty function" that's some sort of sum of errors, and use standard optimization techniques to minimize it. The trouble is, these approaches tend to work spottily -- in particular, they sometimes pick out local rather than global optima (so that the error can be quite high after all.)
The approach in the paper I linked is different. We break the graph into overlapping smaller subgraphs, so small that they can only be embedded in one way (that's called rigidity) and then "stitch" them together consistently. The "stitching" is done with a very handy trick involving eigenvectors of sparse matrices. But the point I want to emphasize here is that you have to look at the small scale, and let all the little patches embed themselves as they like, before trying to reconcile them globally.
Now, rather daringly, I want to apply this idea to ethics. (This is an expansion of a post people seemed to like: http://lesswrong.com/lw/1xa/human_values_differ_as_much_as_values_can_differ/1y )
The thing is, human values differ enormously. The diversity of values is an empirical fact.
The Japanese did not have a word for "thank you" until the Portuguese gave them one; this is a simple example, but it absolutely shocked me, because I thought "thank you" was a universal concept. It's not.(edited for lack of fact-checking.) And we do not all agree on what virtues are, or what the best way to raise children is, or what the best form of government is. There may be no principle that all humans agree on -- dissenters who believe that genocide is a good thing may be pretty awful people, but they undoubtedly exist. Creating the best possible world for humans is a synchronization problem, then -- we have to figure out a way to balance values that inevitably clash. Here, nodes are individuals, each individual is tied to its neighbors, and a choice of embedding is a particular action. The worse the embedding near an individual fits the "true" underlying manifold, the greater the "penalty function" and the more miserable that individual is, because the action goes against what he values.If we can extend the metaphor further, this is a problem for utilitarianism. Maximizing something globally -- say, happiness -- can be a dead end. It can hit a local maximum -- the maximum for those people who value happiness -- but do nothing for the people whose highest value is loyalty to their family, or truth-seeking, or practicing religion, or freedom, or martial valor. We can't really optimize, because a lot of people's values are other-regarding: we want Aunt Susie to stop smoking, because of the principle of the thing. Or more seriously, we want people in foreign countries to stop performing clitoridectomies, because of the principle of the thing. And Aunt Susie or the foreigners may feel differently. When you have a set of values that extends to the whole world, conflict is inevitable.
The analogue to breaking down the graph is to keep values local. You have a small star-shaped graph of people you know personally and actions you're personally capable of taking. Within that star, you define your own values: what you're ready to cheer for, work for, or die for. You're free to choose those values for yourself -- you don't have to drop them because they're perhaps not optimal for the world's well-being. But beyond that radius, opinions are dangerous: both because you're more ignorant about distant issues, and because you run into this problem of globally reconciling conflicting values. Reconciliation is only possible if everyone's minding their own business. If things are really broken down into rigid components. It's something akin to what Thomas Nagel said against utilitarianism:
"Absolutism is associated with a view of oneself as a small being interacting with others in a large world. The justifications it requires are primarily interpersonal. Utilitarianism is associated with a view of oneself as a benevolent bureaucrat distributing such benefits as one can control to countless other beings, with whom one can have various relations or none. The justifications it requires are primarily administrative." (Mortal Questions, p. 68.)
Anyhow, trying to embed our values on this dark continent of a manifold seems to require breaking things down into little local pieces. I think of that as "cultivating our own gardens," to quote Candide. I don't want to be so confident as to have universal ideologies, but I think I may be quite confident and decisive in the little area that is mine: my personal relationships; my areas of expertise, such as they are; my own home and what I do in it; everything that I know I love and is worth my time and money; and bad things that I will not permit to happen in front of me, so long as I can help it. Local values, not global ones.
Could any AI be "friendly" enough to keep things local?