I may well have overgeneralized. I was basing the idea on remembering an essay saying that FAI should be designed to be non-sentient, and seeing concerns about how uploads and simulated people would be treated.
Yes, that 'have' vs 'can have' distinction changes everything---but most people are less picky with general claims than I.
I suppose that moral concern would apply to any of sentient programs that humans would be likely to create.
Food for thought. I wouldn't rule this out as a possibility and certainly the proportion of 'morally relevant' programs in this group skyrockets over the broader class. I'm not too sure what we are likely to create. How probable is it that we succeed at creating sentience but fail at FAI?
I think creating sentience is a much easier project than FAI, especially proven FAI. We've got plenty of examples of sentience.
Creating sentience which isn't much like the human model seems very difficult-- I'm not even sure what that would mean, with the possible exception of basing something on cephalopods. OK, maybe beehives are sentient, too. How about cities?
In this video, long about 48:00, Eliezer talks about uploading and about how it wouldn't be murder if his meat body were anesthetized before the upload and killed without regaining consciousness.
It's arguable that it wouldn't be murder, but I'm not clear about why Eliezer would want to do it that way. I've got some guesses about why one might want to not let the meat body wake up (legal and practical complications of a double but diverging identity, the meat version feeling hopelessly envious), but I'm not sure whether either of them apply.
On the other hand, I can think of a couple of reasons for *not* eliminating the meat version-- one is that two Eliezers would presumably be better than one, though I don't have a strong intuition about the optimum number of Eliezers. The other, which I consider to be more salient, is that the meat version is a backup in case the upload isn't as good as hoped.
More generally, what would folks here consider to be good enough evidence that uploading was worth doing?