How did it not go far enough? What would you like me to add?
even if they were just sitting in a box for eternity.
The could be super fulfilled doing other things as well. Some people (I think EY is included in this group) wouldn't want to just sit in a box for eternity. However, they could still be super fulfilled by altering their hedonic set-point digitally.
According to that view, making a FAI would be a HUGE missed opportunity, since it wouldn't do that.
There were too many pronouns for me to understand what you were talking about. Which view? And what wouldn't the FAI do? Generally when I hear missed opportunity I think of something that you don't do, but you should have done. So I don't really understand how making an FAI is a missed opportunity.
even if they were just sitting in a box for eternity.
The could be super fulfilled doing other things as well.
Right, but if they can be just as fulfilled in the box, and it allows more humans to be in this simulated utopia, why not stack 'em in boxes?
If this sounds bad, it's because it's horrible from a utility-maximizing standpoint. Utility maximizers use their own standards when evaluating the future, not the standards of the beings of the time. And so when you hear "stack 'em in boxes," the utility maximizer part of you goes "that ...
If you were a utilitarian, then why would you want to risk creating an AGI that had the potential to be an existential risk, when you could eliminate all suffering with the advent of WBE (whole brain emulation) and hence virtual reality (or digital alteration of your source code) and hence utopia? Wouldn't you want to try to prevent AI research and just promote WBE research? Or is it that AGI is more likely to come before WBE and so we should focus our efforts on making sure that the AGI is friendly? Or maybe uploading isn't possible for technological or philosophical reasons (substrate dependence)?
Is there a link to a discussion on this that I'm missing out on?