Well, then, so much for the following:
No. We ruled out perfect quantum-level copies, which does not rule out near-copies being morally relevant.
For example, you plus one second is completely different on the quantum level due to uninteresting things like thermal noise, but they are just as important as you are. Likewise, a molecule-level copy of you would be pretty much the same (at least physically) as movement through time (to within the tolerances we normally deal with), as would, I suspect, cell-level plus a few characterization parameters like "connection strength" and "activation level", and I bet you could even drop activation level (as happens in sleep) and many low-level details like exact cell arrangement.
Basically, humans are not exactly isolated 0.01 Kelvin quantum computers (and even they have decoherence time much less than a second), so if you want exact continuity, you already don't have it. You have to generalize your moral intuitions to approximate continuity just to accept normal things like existing at 310 K, sneezes destroying thousands of brain cells, sleep rebooting everything, and random chemicals influencing your cognition. Many people who do so decide that the details of the computational substrate don't really matter; it's the high level behaviors that matter. Hence patternism.
I'd be interested in whether you still disagree and why.
I'd be interested in whether you still disagree and why.
I'm still trying to figure out what does and does not matter to my concept of continuity and why.
Let me ask you this: which person from five seconds in the future do you care more about protecting: yourself or a random person similar to you?
Is there some theoretical threshold of similarity between yourself and another person beyond which you will no longer be sure which location will be occupied by the brain-state your current brain-state will evolve to in the next second?
It seems to me that most AI researchers on this site are patternists in the sense of believing that the anti-zombie principle necessarily implies:
1. That it will ever become possible *in practice* to create uploads or sims that are close enough to our physical instantiations that their utility to us would be interchangeable with that of our physical instantiations.
2. That we know (or will know) enough about the brain to know when this threshold is reached.
But, like any rationalists extrapolating from unknown unknowns... or heck, extrapolating from anything... we must admit that one or both of the above statements could be wrong without also making friendly AI impossible. What would be the consequences of such error?
I submit that one such consequence could be an FAI that is also wrong on these issues but not only do we fail to check for such a failure mode, it actually looks to us like what we would expect the right answer to look because we are making the same error.
If simulation/uploading really does preserve what we value about our lives then the safest course of action is to encourage as many people to upload as possible. It would also imply that efforts to solve the problem of mortality by physical means will at best be given an even lower priority than they are now, or at worst cease altogether because they would seem to be a waste of resources.
Result: people continue to die and nobody including the AI notices, except now they have no hope of reprieve because they think the problem is already solved.
Pessimistic Result: uploads are so widespread that humanity quietly goes extinct, cheering themselves onward the whole time
Really Pessimistic Result: what replaces humanity are zombies, not in the qualia sense but in the real sense that there is some relevant chemical/physical process that is not being simulated because we didn't realize it was relevant or hadn't noticed it in the first place.
Possible Safeguards:
* Insist on quantum level accuracy (yeah right)
* Take seriously the general scenario of your FAI going wrong because you are wrong in the same way and fail to notice the problem.
* Be as cautious about destructive uploads as you would be about, say, molecular nanotech.
* Make sure you knowledge of neuroscience is at least as good as you knowledge of computer science and decision theory before you advocate digital immortality as anything more than an intriguing idea that might not turn out to be impossible.