I can't presume to answer for Eliezer, but I don't think he's yet claimed to know how the brain works. He's also paid considerable attention to the nonsensical nature of some attempts to say that we might "already" know how- IE "emergence", "complexity", and other non-explanations. I'd go so far as to say that it follows directly from the fact that we can't make our own brains from first principles that we don't really understand the ones currently in circulation.
That said, it would be a serious defiance of all precedent if brains somehow had a magical, non-reducible quality by which they refused to comply with empirical observation. It's true that the past success of such study can't reliably predict future trends. By the same logic, however, we can't expect gravity to continue in the future because past trends and consistency are of a different substance than future ones. Until gravity and reductionism actually do give out, we can say reasonably well that gravity is likely to continue and things are likely to be explicable. Following this line of reasoning - that the past may not predict the future at all - could easily kill any plotted course of action relying upon gravity or causality equally well, so why apply it only to cognitive science?
As pertains to brains, we have reasonable inferences that the mind is strictly anchored in a physical substance. Among the oldest I'm aware of is Heraclitis' observation that hitting someone in the head causes stupor, confusion, etc, so the mind probably resides there. More modern versions can include research into brain lesions, neurotransmitters, psychoactive drugs, and the like if you prefer. The only way I can imagine to actually rule out a purely "physical" brain, especially against the weight of current evidence, would be if we could finally map the brain to perfection, watch all the computation it's carrying out, understand it all- and still demonstrate that there's a mysterious magic term in the input or output that definitely comes from nowhere at all. It sounds ridiculous spelled out this way, but that's essentially what postulating "non-reducibility" comes down to- that monitoring an entire brain physically, you could actually watch things come out of nowhere. Certain physicists would find this kind of disturbing, for one.
Additionally, "God did it" and "Energy is conserved" are not isomorphic. One explains nothing; it does not provide any way to plot future events, assuming causality and a fairly stable universe. The other one does provide a way to plot future events, assuming causality and a fairly stable universe. Again, if you want to chuck out causality and a fairly stable universe, I have to wonder why you bother finishing sentences seeing as sound and information propagation are bound to stop working at any time. If we can agree that causality and stability are to remain in play, however, it follows that certain models will correspond to predictable reality and others will not. Going against this doesn't just undermine AI or cognitive science, it actually undermines empiricism in general, which is funny because empiricism has a pretty good track record in spite of it.
Back to our original topic: Reductionism, which (in case you've forgotten) is part of a sequence on the Mind Projection Fallacy. There can be emotional problems in accepting reductionism, if you think that things have to be fundamental to be fun. But this position commits us to never taking joy in anything more complicated than a quark, and so I prefer to reject it.
To review, the reductionist thesis is that we use multi-level models for computational reasons, but physical reality has only a single level. If this doesn't sound familiar, please reread "Reductionism".
Today I'd like to pose the following conundrum: When you pick up a cup of water, is it your hand that picks it up?
Most people, of course, go with the naive popular answer: "Yes."
Recently, however, scientists have made a stunning discovery: It's not your hand that holds the cup, it's actually your fingers, thumb, and palm.
Yes, I know! I was shocked too. But it seems that after scientists measured the forces exerted on the cup by each of your fingers, your thumb, and your palm, they found there was no force left over—so the force exerted by your hand must be zero.
The theme here is that, if you can see how (not just know that) a higher level reduces to a lower one, they will not seem like separate things within your map; you will be able to see how silly it is to think that your fingers could be in one place, and your hand somewhere else; you will be able to see how silly it is to argue about whether it is your hand picks up the cup, or your fingers.
The operative word is "see", as in concrete visualization. Imagining your hand causes you to imagine the fingers and thumb and palm; conversely, imagining fingers and thumb and palm causes you to identify a hand in the mental picture. Thus the high level of your map and the low level of your map will be tightly bound together in your mind.
In reality, of course, the levels are bound together even tighter than that—bound together by the tightest possible binding: physical identity. You can see this: You can see that saying (1) "hand" or (2) "fingers and thumb and palm", does not refer to different things, but different points of view.
But suppose you lack the knowledge to so tightly bind together the levels of your map. For example, you could have a "hand scanner" that showed a "hand" as a dot on a map (like an old-fashioned radar display), and similar scanners for fingers/thumbs/palms; then you would see a cluster of dots around the hand, but you would be able to imagine the hand-dot moving off from the others. So, even though the physical reality of the hand (that is, the thing the dot corresponds to) was identical with / strictly composed of the physical realities of the fingers and thumb and palm, you would not be able to see this fact; even if someone told you, or you guessed from the correspondence of the dots, you would only know the fact of reduction, not see it. You would still be able to imagine the hand dot moving around independently, even though, if the physical makeup of the sensors were held constant, it would be physically impossible for this to actually happen.
Or, at a still lower level of binding, people might just tell you "There's a hand over there, and some fingers over there"—in which case you would know little more than a Good-Old-Fashioned AI representing the situation using suggestively named LISP tokens. There wouldn't be anything obviously contradictory about asserting:
because you would not possess the knowledge
None of this says that a hand can actually detach its existence from your fingers and crawl, ghostlike, across the room; it just says that a Good-Old-Fashioned AI with a propositional representation may not know any better. The map is not the territory.
In particular, you shouldn't draw too many conclusions from how it seems conceptually possible, in the mind of some specific conceiver, to separate the hand from its constituent elements of fingers, thumb, and palm. Conceptual possibility is not the same as logical possibility or physical possibility.
It is conceptually possible to you that 235757 is prime, because you don't know any better. But it isn't logically possible that 235757 is prime; if you were logically omniscient, 235757 would be obviously composite (and you would know the factors). That that's why we have the notion of impossible possible worlds, so that we can put probability distributions on propositions that may or may not be in fact logically impossible.
And you can imagine philosophers who criticize "eliminative fingerists" who contradict the direct facts of experience—we can feel our hand holding the cup, after all—by suggesting that "hands" don't really exist, in which case, obviously, the cup would fall down. And philosophers who suggest "appendigital bridging laws" to explain how a particular configuration of fingers, evokes a hand into existence—with the note, of course, that while our world contains those particular appendigital bridging laws, the laws could have been conceivably different, and so are not in any sense necessary facts, etc.
All of these are cases of Mind Projection Fallacy, and what I call "naive philosophical realism"—the confusion of philosophical intuitions for direct, veridical information about reality. Your inability to imagine something is just a computational fact about what your brain can or can't imagine. Another brain might work differently.