I would argue that if someone constructed an automaton that behaved exactly like I would in any given real-world situation -- including novel situations, which Disney automatons can't handle -- then that automaton would, for all intents and purposes, be as conscious as I am. In fact, this automaton would, in fact, be a copy of me.
Let's imagine that tonight, while you sleep, evil aliens replace everyone else in your home town (except for yourself, that is) with one of those perfect automatons. Would you be able to tell that this had occurred ? If so, how would you determine this ?
Perhaps I might not know the difference, but I am not the only observer here. Would the people replaced know the difference?
Fooling you by replacing me is one thing. Fooling me by replacing me is an entirely more difficult thing to do.
I encounter many intelligent people (not usually LWers, though) who say that despite our recent scientific advances, human consciousness remains a mystery and currently intractable to science. This is wrong. Empirically distinguishable theories of consciousness have been around for at least 15 years, and the data are beginning to favor some theories over others. For a recent example, see this August 2011 article from Lau & Rosenthal in Trends in Cognitive Sciences, one of my favorite journals. (Review articles, yay!)
Abstract: