Thanks, I'll look into it.

(And all my ranting still didn't address the fundamental difficulty: There is no rational way to choose from among different projections of values held by multiple agents, projections such as Rawlsianism and utilitarianism.)

Although what if we told each party to submit goals rather than non-goal preferences? If the AI has access to a model specifying which actions lead to which consequences, then it can search for those actions that maximize the number of goals fulfilled regardless of which party submitted them, or perhaps takes a Rawlsian approach of trying to maximize the number of goals fulfilled that were submitted by whichever party will have the least number of goals fulfilled if that sequence of actions were taken, etc. That seems very imaginable to me. You can then have heuristics that constrain the search space and stuff. You can also have non-goal preferences in addition to goals if the parties have any of those.

In that light, it seems to me that the problem was inferring goals from a set of preferences which were not purely non-goal preferences but were actually presented with some unspecified goals in mind. Eg. One party wanted chocolate, but said, "I want to go to the store" instead. If that was the source of the original problem, then we can see why we might need an AI to solve it, since it calls for some lightweight mind reading. Of course, a CEV-implementing AI would have to be a mind reader anyway, since we don't really know what our goals ultimately are given everything we could know about reality.

This still does not guarantee basic morality, but parties should at least recognize some of their ultimate goals in the end result. They might still grumble about the result not being exactly what they wanted, but we can at least scold them for lacking a spirit of compromise.

All this presupposes that enough of our actions can be reduced to ultimate goals that can be discovered, and I don't think this process guarantees we will be satisfied with the results. For example, this might erode personal freedom to an unpleasant degree. If we would choose to live in some world X if we were wiser and nicer than we are, then it doesn't necessarily follow that X is a Nice Place to Live as we are now. Changing ourselves to reach that level of niceness and wisdom might require unacceptably extensive modifications to our actual selves.

Thanks. That makes sense to me.

The real difficulty is that when you combine two sets of preferences, each of which make sense on their own, you get a set of preferences that makes no sense whatsoever: http://plato.stanford.edu/entries/economics/#5.2 https://www.google.com/search?q=site%3Aplato.stanford.edu+social+choice&ie=utf-8&oe=utf-8

There is no easy way to resolve this problem. There is also no known method that takes such an inconsistent set of preferences as input and gives a consistent set of preferences as output such that the output would be recognizable to either party who contributed an original set of preferences as furthering any of their original goals. These random decisions are required so often in cases where there isn't an unanimous agreement that in practice, there would be a large component of arbitrariness every single time CEV tries to arrive at a uniform set of preferences by extrapolating volitions of multiple agents into the future.

This doesn't mean the problem is unresolvable, just that it's an AI problem in its own right, but given these problems, wouldn't it be better to pick whichever Nice Place to Live is the safest to reach instead of bothering with CEV? I say this because I'm not sure Nice Place to Live can be defined in terms of CEV, as any CEV-approved output. Because of the preference aggregation problem, I'm not certain that a world that is provably CEV-abiding also provably avoids flagrant immorality. Two moral frameworks when aggregated by a non-smart algorithm might give rise to an immoral framework, so I'm not sure the essence of the problem is resolved just by CEV as explained in the paper.

Interesting. In that case, would you say an AI that provably implements CEV's replacement is, for that reason, provably Friendly? That is, AIs implementing CEV's replacement form an analytical subset of Friendly AIs? What is the current replacement for CEV anyway? Having some technical material would be even better. If it's open to the public, then I'd like to understand how EY proposes to install a general framework similar to CEV at the "initial dynamic" stage that can predictably generate a provably Friendly AI without explicitly modeling the target of its Friendliness.

That would seem to be the best possible solution, but I have never heard aeroplane engineers claim that their designs are "provably airworthy". If you take the aeroplane design approach, then isn't "provably Friendly" a somewhat misleading claim to make, especially when you're talking about pushing conditions to the extreme that you yourself admit are beyond your powers of prediction? The aeroplane equivalent would be like designing a plane so powerful that its flight changes the atmospheric conditions of the entire planet, but then the plane uses a complicated assembly of gyroscopes or something to continue flying in a straight line. However, if you yourself cannot predict which specific changes the flight of the plane will make, then how can you claim that you can prove that particular assembly of gyroscopes is sufficient to keep the plane on the preplanned path? On the other hand, if you can prove which specific changes the plane's flight will make that are relevant to its flight, then you have a mathematical definition of the target atmosphere at a sufficient depth of resolution to design such an assembly. Does MIRI think it can come up with an equivalent mathematical model of humanity with respect to AI?

The SEP says that preferences cannot be aggregated without additional constraints on how the aggregation is to be done, and the end result changes depending on things like the order of aggregation, so these additional constraints take on the quality of arbitrariness. How does CEV get around that problem?

Oh well, I agree with the English that Kant was a romantic philosopher. Rousseau was a primary source of inspiration for him. (I agree with Dewey that writers (Goethe) and philosophers (Kant) give expression to popular views more than shaping them. OTOH, as much as I admire Goethe, I think Oswald Spengler went too far in trying to interpret him as a universal philosopher.)

"In the 19th century the German idea was about not having wars between German states," is a true statement, but it leaves out crucial details. For example, there are many people who agree that European nations should not war against each other, but are bitterly critical of the details of how that general plan was implemented in practice.

I think it follows that the European idea is not reducible to the notion that European states should not fight. If you do not agree, then I apologize for using terms like "European idea" and "German idea" in a sense you didn't intend, but my point can be easily reworded using "implementation of the German idea" in place of "German idea".

The point I'm trying to make is that, like I said, Germany is currently outcompeting the Anglo-American world on the terms of the Anglo-American world, not on the original terms of Germany. Arguably, England wanted to end European wars in the 19th century as well. Who would you say got their way in the end, England or Germany?

Look, the collapse of a state is the collapse of state regardless of ideological roles. (Modern Germany is fundamentally Anglo-American in design and very successful. That is the point, since you were citing the success of contemporary Germany.)

(...Nah, it would take far too long to discuss the state of Germany prior to WWI.)

Faust really was a central figure in the German idea, I'm afraid. I don't know how consciously Goethe was complicit in this, and this has nothing to do with what he would have had problems for saying what when he published Faust.

Of course Faust is still a part of German culture. He's part of world culture, a typically German vision of the universal man. (I am personally a huge fan of Faust.)

I don't understand the contradiction in saying that X and Y have different wishes for the future owing to cultural differences. (And I don't understand what Habermas' Europe has to do with the 19th century German idea. Habermas has openly stated that the German intellectual tradition is inadequate for criticizing fascism and consciously borrowed from Anglophone thinkers. The most striking difference between thinkers who have gained a standing in the Anglophone world and thinkers from the rest of the world is their careful, deliberate anti-existentialism.)

(Kant is commonly admitted to be a romantic philosopher, and I found this link: http://philosophyisnotaluxury.com/2010/08/12/romanticism-and-existential-philosophy/)