Yvain: It must have something to do with capitalism including a term for the human utility function in the form of demand

Why, yes, I do think that has something to do with why the market builds houses with air conditioning instead of tiny little cells.

Hal: Do these considerations offer useful insights for the average person living his life? Or are they just abstract philosophy without practical import for most people?

Well, this particular abstract philosophy could end up having a pretty large practical import for all people, if they end up reprocessed into paperclips. But to answer the intent of your question, hence the whole extension to general optimism as a special case of anthropomorphism.

Robin: To get this whole line of reasoning off the ground, you need a decent way to rank phenomena in terms of how similar they are to us. Given this ranking, the warning is to beware of treating low ranked items like high ranked items. On AI, you need to give an argument why AI is a low ranked item, i.e., why AI is especially unlike us.

Name me any high-ranked item that does not share causal parentage with a human. Chimps, for example, are worthy objects of anthropomorphism - and 95% genetically similar to us due to common ancestry.

Pearson: I have a question for you Eliezer. When you were figuring out how powerful AIs made from silicon were likely to be, did you have a goal that you wanted? Do you want AI to be powerful so it can stop death?

I think I was pretty much raised believing in the intelligence explosion (i.e. read "Great Mambo Chicken and the Transhuman Condition" before puberty). As a teenager I thought it was likely that AIs would be able to violate what our civilization believes to be the laws of physics, and e.g. enable interstellar travel at FTL speeds. As I grew up and my knowledge became more constraining, and intelligence began to seem less like magic and more like a phenomenon within physics, it became much less absurd to think that an SI might still be constrained by the lightspeed limit we know - especially given the Fermi Paradox. (Of course I do still assign a fair probability that we are very far from knowing the final laws of physics - I would bet at >50% on an SI being able to accomplish in practice at least one thing we deem "physically impossible".)

If you live with physics as we know it, that does seem to imply no immortality - just living for a very long time, and then dying. Though I still hold out hope.

So the answer to your question's intent is essentially "Yes" on both counts; and I have grown less confident of my hopes, and less awed, over time. But such trivial and physically possible deeds as building molecular nanotechnology, or thinking a million times as fast as a human, I am still fairly confident about.

Prase: When we remove whole groups from the gene pool because of some group characteristic (i.e. averaged over the population of that group), it sounds for me natural to call that a group selection.

It is; the question is whether such group selection can overcome a countervailing individual selection pressure. Mathematically, this requires group selection pressure to be extremely strong, or individual selection pressures to be very weak, or both.

A "group-selected" characteristic would be one produced by selection on the level of groups, such as cannibalism in Michael Wade's experiment. Not a characteristic that is "nice toward the group" according to a sense of human aesthetics. Although cannibalism does help the group, if high-population groups are regularly eliminated. And in fact this characteristic was produced by group selection; it was a group-fitness-increasing adaptation for population control. Cannibalism from individual selection pressures was much weaker, in the control groups. It's just not the way that you or I would think of helping.