One of the problems here is that of using our intuition on consciousness as a guide to processes well outside our experience. Why should we believe our common-sense intuition on whether a computer has consciousness, or whether a pencil and paper simulation has consciousness when both are so far beyond our actual experience? It's like applying our common sense understanding of physics to the study of atoms, or black holes. There's no reason to assume we can extrapolate that far intuitively with any real chance of success.

Upvoted. I'm stealing this for use in future off-LW discussions of consciousness. ;-)