What I do think is that the humans are not acting according to their `real' preferences, and that they would realize this if they understood Eliezer's arguments.
Human real preferences aren't utility based, not even close, and this is a big potential problem. So they have to make their preferences closer to a utility function, using some methods or other. But humans never should act according to their messy 'real' preferences.
What moral status do you attach to humans who do not currently exist, but definitely will exist in the future?
Same as I do to people today. Simple heuristic: any choice that causes increased utility to any agent that exists at any time is always positive - giving a dollar to somebody in two generation is good, whoever they are.
On the other hand, choices that increase or decrease the number of agents - giving birth to that person in two generations or not - are more complicated.
Good luck!
Thanks!
Have you seen http://meteuphoric.wordpress.com/2011/03/13/if-birth-is-worth-nothing-births-are-worth-anything/ ? It may help you notice any inconsistencies between possible utility functions and your values.
Some people see never-existed people as moral agents, and claim that we can talk about their preferences. Generally this means their personal preference in existing versus non-existing. Formulations such "it is better for someone to have existed than not" reflect this way of thinking.
But if the preferences of never-existed are relevant, then their non-personal perferences are also relevant. Do they perfer a blue world or a pink one? Would they want us to change our political systems? Would they want us to not bring into existence some never-existent people they don't like?
It seems that those who are advocating bringing never-existent people into being in order to satisfy those people's preferences should be focusing their attention on their non-personal preferences instead. After all, we can only bring into being so many trillions of trillions of trillions; but there is no theoretical limit to the number of never-existent people whose non-personal preferences we can satisfy. Just get some reasonable measure across the preferences of never-existent people, and see if there's anything that sticks out from the mass.