Dan - We have different projects; I'm not trying to "fix the counterintuitiveness of consciousness." I'm interested in whether it is in principle susceptible to physical reduction. (We can answer these sorts of questions by understanding alone. I don't need to do science in order to appreciate the conditional that if physical investigation reveals particles that play such-and-such a role, then objects such as hands will be reducible to said arrangements of particles. There is no coherently conceivable 'hand-zombie' world, analogous to the phenomenal-zombie world, that is identical to ours in all matters of fundamental physics, but somehow lacking in hands.)
Incidentally, I agree that the human brain is capable of amazing computational feats. There are many aspects of its physical functioning that I don't yet understand, though I'm confident future science will make further progress here. None of that is relevant to the present discussion.
Yeah, our projects are actually exactly the same. We're just using completely different tools and/or words. For instance, I am still waiting to figure out how "understanding", "natural law" and "phenomenal" things can be given meaningful weight above voodoo, karmic rebirth, or dreamtime. Being better tailored to our collective cultural background than the alternatives would be is about all it seems to have, as is.
I'm also baffled by the statement that a question about whether the mind can be reduced to the brain could not be informed by the science of how the brain works. Really? I need your tools. I would love to know how things work without having to first understand how they work. I'd also like to mention that the zombie thought experiment has its conclusion embedded in its premises. In order to accept that such zombies are logically coherent, you need to already think that "qualia" have some magical nonphysical quality. It's the definition of circular reasoning, and there's nothing to establish its validity over "hand-zombies" except that it's more appealing to the person thinking it.
Back to our original topic: Reductionism, which (in case you've forgotten) is part of a sequence on the Mind Projection Fallacy. There can be emotional problems in accepting reductionism, if you think that things have to be fundamental to be fun. But this position commits us to never taking joy in anything more complicated than a quark, and so I prefer to reject it.
To review, the reductionist thesis is that we use multi-level models for computational reasons, but physical reality has only a single level. If this doesn't sound familiar, please reread "Reductionism".
Today I'd like to pose the following conundrum: When you pick up a cup of water, is it your hand that picks it up?
Most people, of course, go with the naive popular answer: "Yes."
Recently, however, scientists have made a stunning discovery: It's not your hand that holds the cup, it's actually your fingers, thumb, and palm.
Yes, I know! I was shocked too. But it seems that after scientists measured the forces exerted on the cup by each of your fingers, your thumb, and your palm, they found there was no force left over—so the force exerted by your hand must be zero.
The theme here is that, if you can see how (not just know that) a higher level reduces to a lower one, they will not seem like separate things within your map; you will be able to see how silly it is to think that your fingers could be in one place, and your hand somewhere else; you will be able to see how silly it is to argue about whether it is your hand picks up the cup, or your fingers.
The operative word is "see", as in concrete visualization. Imagining your hand causes you to imagine the fingers and thumb and palm; conversely, imagining fingers and thumb and palm causes you to identify a hand in the mental picture. Thus the high level of your map and the low level of your map will be tightly bound together in your mind.
In reality, of course, the levels are bound together even tighter than that—bound together by the tightest possible binding: physical identity. You can see this: You can see that saying (1) "hand" or (2) "fingers and thumb and palm", does not refer to different things, but different points of view.
But suppose you lack the knowledge to so tightly bind together the levels of your map. For example, you could have a "hand scanner" that showed a "hand" as a dot on a map (like an old-fashioned radar display), and similar scanners for fingers/thumbs/palms; then you would see a cluster of dots around the hand, but you would be able to imagine the hand-dot moving off from the others. So, even though the physical reality of the hand (that is, the thing the dot corresponds to) was identical with / strictly composed of the physical realities of the fingers and thumb and palm, you would not be able to see this fact; even if someone told you, or you guessed from the correspondence of the dots, you would only know the fact of reduction, not see it. You would still be able to imagine the hand dot moving around independently, even though, if the physical makeup of the sensors were held constant, it would be physically impossible for this to actually happen.
Or, at a still lower level of binding, people might just tell you "There's a hand over there, and some fingers over there"—in which case you would know little more than a Good-Old-Fashioned AI representing the situation using suggestively named LISP tokens. There wouldn't be anything obviously contradictory about asserting:
because you would not possess the knowledge
None of this says that a hand can actually detach its existence from your fingers and crawl, ghostlike, across the room; it just says that a Good-Old-Fashioned AI with a propositional representation may not know any better. The map is not the territory.
In particular, you shouldn't draw too many conclusions from how it seems conceptually possible, in the mind of some specific conceiver, to separate the hand from its constituent elements of fingers, thumb, and palm. Conceptual possibility is not the same as logical possibility or physical possibility.
It is conceptually possible to you that 235757 is prime, because you don't know any better. But it isn't logically possible that 235757 is prime; if you were logically omniscient, 235757 would be obviously composite (and you would know the factors). That that's why we have the notion of impossible possible worlds, so that we can put probability distributions on propositions that may or may not be in fact logically impossible.
And you can imagine philosophers who criticize "eliminative fingerists" who contradict the direct facts of experience—we can feel our hand holding the cup, after all—by suggesting that "hands" don't really exist, in which case, obviously, the cup would fall down. And philosophers who suggest "appendigital bridging laws" to explain how a particular configuration of fingers, evokes a hand into existence—with the note, of course, that while our world contains those particular appendigital bridging laws, the laws could have been conceivably different, and so are not in any sense necessary facts, etc.
All of these are cases of Mind Projection Fallacy, and what I call "naive philosophical realism"—the confusion of philosophical intuitions for direct, veridical information about reality. Your inability to imagine something is just a computational fact about what your brain can or can't imagine. Another brain might work differently.