Max, as you can see from Eliezer's reply, MIRI people (and other FAI proponents) are largely already aware of the problems you brought up in your paper. (Personally I think they are still underestimating the difficulty of solving those problems. For example, Peter de Blanc and Eliezer both suggest that humans can already solve ontological crises, implying that the problem is merely one of understanding how we do so. However I think humans actually do not already have such an ability, at least not in a general form that would be suitable for implementing in a Friendly AI, so this is really a hard philosophical problem rather than just one of reverse engineering.)

Also, you may have misunderstood why Nick Bostrom talks about "goal retention" in his book. I think it's not meant to be an argument in favor of building FAI (as you suggest in the paper), but rather an argument for AIs being dangerous in general, since they will resist attempts to change their goals by humans if we realize that we built AIs with the wrong final goals.

Thanks Eliezer for your encouraging words and for all these interesting comments! I agree with your points, and we clearly agree on the bottom line as well: 1) Building FAI is hard and we’re far from there yet. Sorting out “final goal” issues is part of the challenge. 2) It’s therefore important to further research these questions now, before it’s too late. :-)

This should be awesome, except for the 2-minute introduction that will be given by this annoying Swedish guy (me). Be there for be square! ;-)