Don't think in terms of choosing what value to assign, think in terms of figuring out what value your utility functions already assigns to it...

I don't think that is possible. Consider the difference between a hunter-gatherer, who cares about his hunting success and to become the new clan chief, and a member of lesswrong who wants to determine if a "sufficiently large randomized Conway board could turn out to converge to a barren 'all off' state."

The utility of the success in hunting down animals and proving abstract conjectures about cellular automata is largely determined by factors such as your education, culture and environmental circumstances. The same hunter gatherer who cared to kill a lot of animals, to get the best ladies in its clan, might have under different circumstances turned out to be a vegetarian mathematicians solely caring about his understanding of the nature of reality. Both sets of values are to some extent mutually exclusive or at least disjoint. Yet both sets of values are what the person wants, given the circumstances. Change the circumstances dramatically and you change the persons values.

You might conclude that what the hunter-gatherer really wants is to solve abstract mathematical problems, he just doesn't know about that. But there is no set of values that a person really wants. Humans are largely defined by the circumstances they reside in. If you already knew a movie, you wouldn't watch it. To be able to get your meat from the supermarket changes the value of hunting.

If "we knew more, thought faster, were more the people we wished we were, and had grown up closer together" then we would stop to desire what we learnt, wish to think even faster, become even different people and get bored of and rise up from the people similar to us.