You assume that the utility of two different worlds can not be exactly equal. edit: or maybe you don't. In any case, this AI which does absolutely nothing in our world is no more useful than AI that does nothing in all possible worlds, or just a brick.

Also, the desire for mankind (and life) not to be straitjacketed, is my view, i'm not sure it is coherently shared by mankind, and in fact i'm not even sure i like the way it is going if it is not straitjacketed in some way. edit: to clarify. I like the heuristics of maximizing the future choices for me. It is part of my values, that i don't want removed. I don't like [consequences of] this heuristic for mankind. Mankind is a meta-organism that is dumb and potentially self destructive.

edit: To clarify. What I am saying, is that there's conflict between two values whose product matters. Survival vs freedom. Survival without freedom is bad. Freedom without survival is nonsense.

Moral Universalism could be true in some sense, but not automatically compelling, and the AI would need to be programmed to find and/or follow it.

My original post had this possibility. Where you make the AI that develops much of the morality (which it would really have to). edit: note that the AI in question may be just a theorem prover which tries to find some universal moral axioms, but is not itself moral or compelled to implement anything in real world.

There could be a uniquely specified human morality that fulfills much of the same purpose Moral Universalism does for humans.

What's in 10 millions years? 100 millions? A straitjacket for intelligent life.

It might be possible to specify what we want in a more dynamic way than freezing in current customs.

We would still want some limits from our values right now, e.g. so that the society wouldn't steer itself to suicide somehow. Even rules like 'it is good if 99% of people agree with it" can steer us into some really nasty futures over the time. Other issue is the possibility of de-evolution of human intelligence. We would not want to lock in all the customs, but some of the values of the today, would get frozen in.

Name 1 then.

edit: and it's not even a dichotomy. There's the hypothetical AIs which implement some moral absolute that is good for all cultures, possible cultures, and everyone, which we would invent, aliens would invent, whatever we evolve into could invent, etc. If those do not exist, then what exists that isn't to some extent culturally specific to h. Sapiens circa today?