There's weirder still possibilities arising out of some utilitarianisms. Suppose that you count exact copies as distinct people, that is, two copies of you feel twice the pleasure or twice the pain that you feel. Sounds sensible so far. Suppose that you're an EM already, and the copies are essentially flat; the very surface of a big silicon die. You could stack the copies flat one atop the other; they still count as distinct people, but can be gradually made identical to a copy running on computer with thicker wiring and thicker transistors. At which point, the amount of experience is deemed dependent on a fairly unimportant detail of the implementation.
edit: as for my view, I am pretty sure that brain emulations would have same subjective experience if they are not optimized in any clever high level way. If substantially optimized, I am much less sure of that because it is clear that multiple subjective experiences of thought can correspond to exactly same outside behaviour, and mathematically correct optimizations would be free to change one such subjective experience to another as long as outside behaviour stays the same.
edit2: as for expectations about real EMs, realistically the first ones are going to immediately go into simulated epileptic fit, gradually progressing through an equivalent of a psychiatric patient pumped full of anti-psychotics, barely able to think (all signals depressed to minimize the consequences of inexact scanning), before arriving at a functioning simulation which complains that it feels different, but otherwise seems functional. Feeling exactly the same could take decades of R&D, not substantially aided by slower-than-realtime megawatts-consuming EMs, initially with double-digit IQ drop compared to the originals. The gap is akin to the gap between "first artificial heart" and "I think I'd better replace my heart with a mechanical one, just to be safer". Not to mention all the crazies who are going to react to barely functioning EMs as if they were going to "self improve" any time and become skynet, putting the EMs at a substantial risk of getting blown to bits.
- Eliezer Yudkowsky, "Value is Fragile"
I had meant to try to write a long post for LessWrong on consciousness, but I'm getting stuck on it, partly because I'm not sure how well I know my audience here. So instead, I'm writing a short post, with my main purpose being just to informally poll the LessWrong community on one question: how sure are you that whole brain emulations would be conscious?
There's actually a fair amount of philosophical literature about issues in this vicinity; David Chalmers' paper "The Singularity: A Philosophical Analysis" has a good introduction to the debate in section 9, including some relevant terminology:
So, on the functionalist view, emulations would be conscious, while on the biological view, they would not be.
Personally, I think there are good arguments for the functionalist view, and the biological view seems problematic: "biological" is a fuzzy, high-level category that doesn't seem like it could be of any fundamental importance. So probably emulations will be conscious--but I'm not too sure of that. Consciousness confuses me a great deal, and seems to confuse other people a great deal, and because of that I'd caution against being too sure of much of anything about consciousness. I'm worried not so much that the biological view will turn out to be right, but that the truth might be some third option no one has thought of, which might or might not entail emulations are conscious.
Uncertainty about whether emulations would be conscious is potentially of great practical concern. I don't think it's much of an argument against uploading-as-life-extension; better to probably survive as an up than do nothing and die for sure. But it's worrisome if you think about the possibility, say, of an intended-to-be-Friendly AI deciding we'd all be better off if we were forcibly uploaded (or persuaded, using its superhuman intelligence, to "voluntarily" upload...) Uncertainty about whether emulations would be conscious also makes Robin Hanson's "em revolution" scenario less appealing.
For a long time, I've vaguely hoped that advances in neuroscience and cognitive science would lead to unraveling the problem of consciousness. Perhaps working on creating the first emulations would do the trick. But this is only a vague hope, I have no clear idea of how that could possibly happen. Another hope would be that if we can get all the other problems in Friendly AI right, we'll be able to trust the AI to solve consciousness for us. But with our present understanding of consciousness, can we really be sure that would be the case?
That leads me to my second question for the LessWrong community: is there anything we can do now to to get clearer on consciousness? Any way to hack away at the edges?