This attacks a straw-man utilitarianism, in which you need to compute precise results and get the one correct answer. Functions can be approximated; this objection isn't even a problem.
Not every function can be approximated efficiently, though. I see the scope of morality as addressing human activity where human activity is a function space itself. In this case the "moral gradient" that the consequentialist is computing is based on a functional defined over a function space. There are plenty of function spaces and functionals which are very hard to efficiently approximate (the Bayes predictors for speech recognition and machine vision fall into this category) and often naive approaches will fail miserably.
I think the critique of utility functions is not that they don't provide meaning, but that they don't necessarily capture the meaning which we would like. The incoherence argument is that there is no utility function which can represent the thing we want to represent. I don't buy this argument mostly because I've never seen a clear presentation of what it is that we would preferably represent, but many people do (and a lot of these people study decision-making and behavior whereas I study speech signals). I think it is fair to point out that there is only a very limited biological theory of "utility" and generally we estimate "utility" phenomenologically by studying what decisions people make (we build a model of utility and try to refine it so that it fits the data). There is a potential that no utility model is actually going to be a good predictor (i.e. that there is some systematic bias). So, I put a lot of weight on the opinions of decision experts in this regard: some think utility is coherent and some don't.
The deontologist's rules seem to do pretty well as many of them are currently sitting in law books right now. They form the basis for much of the morality that parents teach their children. Most utilitarians follow most of them all the time, anyway.
My personal view is to do what I think most people do: accept many hard constraints on one's behavior and attempt to optimize over estimates of projections of a moral gradient along a few dimensions of decision-space. I.e. I try to think about how my research may be able to benefit people, I also try to help out my family and friends, I try to support things good for animals and the environment. These are areas where I feel more certain that I have some sense where some sort of moral objective function points.
What is the justification for the incoherence argument? Is there a reason, or is it just "I won't believe in your utility function until I see it"?
This is a post about moral philosophy, approached with a mathematical metaphor.
Here's an interesting problem in mathematics. Let's say you have a graph, made up of vertices and edges, with weights assigned to the edges. Think of the vertices as US cities and the edges as roads between them; the weight on each road is the length of the road. Now, knowing only this information, can you draw a map of the US on a sheet of paper? In mathematical terms, is there an isometric embedding of this graph in two-dimensional Euclidean space?
When you think about this for a minute, it's clear that this is a problem about reconciling the local and the global. Start with New York and all its neighboring cities. You have a sort of star shape. You can certainly draw this on the plane; in fact, you have many degrees of freedom; you can arbitrarily pick one way to draw it. Now start adding more cities and more roads, and eventually the degrees of freedom diminish. If you made the wrong choices earlier on, you might paint yourself in a corner and have no way to keep all the distances consistent when you add a new city. This is known as a "synchronization problem." Getting it to work locally is easy; getting all the local pieces reconciled with each other is hard.
This is a lovely problem and some acquaintances of mine have written a paper about it. (http://www.math.princeton.edu/~mcucurin/Sensors_ASAP_TOSN_final.pdf) I'll pick out some insights that seem relevant to what follows. First, some obvious approaches don't work very well. It might be thought we want to optimize over all possible embeddings, picking the one that has the lowest error in approximating distances between cities. You come up with a "penalty function" that's some sort of sum of errors, and use standard optimization techniques to minimize it. The trouble is, these approaches tend to work spottily -- in particular, they sometimes pick out local rather than global optima (so that the error can be quite high after all.)
The approach in the paper I linked is different. We break the graph into overlapping smaller subgraphs, so small that they can only be embedded in one way (that's called rigidity) and then "stitch" them together consistently. The "stitching" is done with a very handy trick involving eigenvectors of sparse matrices. But the point I want to emphasize here is that you have to look at the small scale, and let all the little patches embed themselves as they like, before trying to reconcile them globally.
Now, rather daringly, I want to apply this idea to ethics. (This is an expansion of a post people seemed to like: http://lesswrong.com/lw/1xa/human_values_differ_as_much_as_values_can_differ/1y )
The thing is, human values differ enormously. The diversity of values is an empirical fact.
The Japanese did not have a word for "thank you" until the Portuguese gave them one; this is a simple example, but it absolutely shocked me, because I thought "thank you" was a universal concept. It's not.(edited for lack of fact-checking.) And we do not all agree on what virtues are, or what the best way to raise children is, or what the best form of government is. There may be no principle that all humans agree on -- dissenters who believe that genocide is a good thing may be pretty awful people, but they undoubtedly exist. Creating the best possible world for humans is a synchronization problem, then -- we have to figure out a way to balance values that inevitably clash. Here, nodes are individuals, each individual is tied to its neighbors, and a choice of embedding is a particular action. The worse the embedding near an individual fits the "true" underlying manifold, the greater the "penalty function" and the more miserable that individual is, because the action goes against what he values.If we can extend the metaphor further, this is a problem for utilitarianism. Maximizing something globally -- say, happiness -- can be a dead end. It can hit a local maximum -- the maximum for those people who value happiness -- but do nothing for the people whose highest value is loyalty to their family, or truth-seeking, or practicing religion, or freedom, or martial valor. We can't really optimize, because a lot of people's values are other-regarding: we want Aunt Susie to stop smoking, because of the principle of the thing. Or more seriously, we want people in foreign countries to stop performing clitoridectomies, because of the principle of the thing. And Aunt Susie or the foreigners may feel differently. When you have a set of values that extends to the whole world, conflict is inevitable.
The analogue to breaking down the graph is to keep values local. You have a small star-shaped graph of people you know personally and actions you're personally capable of taking. Within that star, you define your own values: what you're ready to cheer for, work for, or die for. You're free to choose those values for yourself -- you don't have to drop them because they're perhaps not optimal for the world's well-being. But beyond that radius, opinions are dangerous: both because you're more ignorant about distant issues, and because you run into this problem of globally reconciling conflicting values. Reconciliation is only possible if everyone's minding their own business. If things are really broken down into rigid components. It's something akin to what Thomas Nagel said against utilitarianism:
"Absolutism is associated with a view of oneself as a small being interacting with others in a large world. The justifications it requires are primarily interpersonal. Utilitarianism is associated with a view of oneself as a benevolent bureaucrat distributing such benefits as one can control to countless other beings, with whom one can have various relations or none. The justifications it requires are primarily administrative." (Mortal Questions, p. 68.)
Anyhow, trying to embed our values on this dark continent of a manifold seems to require breaking things down into little local pieces. I think of that as "cultivating our own gardens," to quote Candide. I don't want to be so confident as to have universal ideologies, but I think I may be quite confident and decisive in the little area that is mine: my personal relationships; my areas of expertise, such as they are; my own home and what I do in it; everything that I know I love and is worth my time and money; and bad things that I will not permit to happen in front of me, so long as I can help it. Local values, not global ones.
Could any AI be "friendly" enough to keep things local?