I think we know of at least one working idea of the utility function for one individual human in theory - show the human all the possible simulated futures and allow them to pick the best.

Of course that would be a complex vector utility function, and you couldn't hope to map that down to a single simple scalar value that aggregates (even with voting).

The true optimal theoretical 'utility function' for humanity as a whole is probably something akin to simulating the evolution of human culture/civilization forward into many possible futures, disseminating all that knowledge, feeding that back in on itself recursively with numerous rounds of re-simulation and then allowing current political/economic systems to evolve forward with all that processed future-path knowledge.