I think what I'm getting towards is there's a difference between human preferences and human preference for other humans. And by human preferences, I mean my own.
That is one objection to Coherent Extrapolated Volition (CEV), i.e. that human values are too diverse. Though the space of possible futures that an AGI could spit out is VERY large compared to the space of futures people would want, even if one takes into consideration the diversity of human values.
I've read through some of the Sequences, but I'm still unclear on a few basic concepts around LW rationality. This is in part to my learning still which benefits from social engagement (ie discussions) rather than just reading. One of those concepts I'm unclear on: Is there an inherent value to human (or sentient) life?
It appears to me that one common theme on this site is that human life (current and future) is very important. Why is that so? Why is the goal people over paper clips?