I think your question again gets right to the nub of the matter. I have no snappy answer to the challenge -here is my long-winded response.
The zombie analogy is a good one. I understand it's meant just as an analogy -the intent is not to fall into the qualia quagmire. The thought is that from a purely naturalistic perspective, people can only properly be seen as, as you put it, preference- or rationality-zombies.
The issue here is the validity of identity claims of the form,
My answer is to compare them to the fate of identity claims relating to sensations (qualia again), such as
Suppose being in pain is found empirically always to correlate to being in brain state DEF, and the identity is proposed. Qualiaphiles will object, saying that this identity misses what's crucial to pain, viz, how it feels. The qualiaphile's thought can be defended by considering the logic of identity claims generally (this adapted from Saul Kripke's Naming and Necessity).
Scientific identity claims are necessary - if water = H2O in this world, then water = H2O in all possible worlds. That is, because water is a natural kind, whatever it is, it couldn't have been anything else. It is possible for water to present itself to us in a different phenomenal aspect ('ice9'!), but this is OK because what's essential to water is its underlying structure, not its phenomenal properties. The situation is different for pain - what's essential to pain is its phenomenal properties. Because pain essentially feels like this (so the story goes), it's correlation with being in brain state DEF can only be contingent. Since identities of this kind, if true, are by their natures necessary, the identity is false.
There is a further step (lots of steps, I admit) to rationality. The thought is that our access to people's rationality is 'direct' in the way our access to pain is. The unmediated judgement of rationality would, if push were to come to shove, trump the scientifically informed, indirect inference from brain states. Defending this proposition would take some doing, but the idea is that we need to understand each other as rational agents before we can get as far as dissecting ourselves to understand ourselves as mere objects.
It is still not clear whether you think rationality is analogous to qualia or is a quale.
On the face of it, there is a tension in adhering both to the idea that there are facts about what it's rational for people to do and to the idea that natural or scientific facts are all the facts there are. The aim of this post is just to try to make clear why this should be so, and hopefully to get feedback on what people think of the tension.
In short
To a first approximation, a belief is rational just in case you ought to hold it; an action rational just in case you ought to take it. A person is rational to the extent that she believes and does what she ought to. Being rational, it is fair to say, is a normative or prescriptive property, as opposed to a merely descriptive one. Natural science, on the other hand, is concerned merely with descriptive properties of things -what they weigh, how they are composed, how they move, and so on. On the face of it, being rational is not the sort of property about which we can theorize scientifically (that is, in the vocabulary of the natural sciences). To put the point another way, rationality concerns what a thing (agent) ought to do, natural science concerns only what it is and will do, and one cannot deduce 'ought' from 'is'.
At greater length
There are at least two is/ought problems, or maybe two ways of thinking about the is/ought problem. The first problem (or way of thinking about the one problem) is posed from a subjective point of view. I am aware that things are a certain way, and that I am disposed to take some course of action, but neither of these things implies that I ought to take any course of action -neither, that is, implies that taking a given course of action would in any sense be right. How do I justify the thought that any given action is the one I ought to take? Or, taking the thought one step further, how, attending only to my own thoughts, do I differentiate merely being inclined to do something from being bound by some kind of rule or principle or norm, to do something?
This is an interesting question -one which gets to the very core of the concept of being justified, and hence of being rational (rational beliefs being justified beliefs). But it isn't the problem of interest here.
The second problem, the problem of interest, is evident from a purely objective, scientific point of view. Consider a lowly rock. By empirical investigation, we can learn its mass, its density, its mineralogical composition, and any number of other properties. Now, left to their own devices, rocks don't do much of anything, comparatively speaking, so it isn't surprising that we don't expect there to be anything it ought to do. In any case, natural science does not imply there is anything it ought to do, I think most will agree.
Consider then a virus particle - a complex of RNA and ancillary molecules. Natural science can tell us how it wiil behave in various circumstances -whether and how it will replicate itself, and so on- but once again surely there is nothing in biochemistry, genetics or other science which implies there is anything our very particle ought to do. It's true that we may think of it as having the goal to replicate itself, and consider it to have made a mistake if it replicates itself inaccurately, but these conceptions do not issue from science. Any sense in which it ought to do something, or is wrong or mistaken in acting in a given way, is surely purely metaphorical (no?).
How about a bacterium? It's orders of magnitude more complicated, but I don't see that matters are any different as regards what it ought to do. Science has nothing to tell us about what if anything is important to a bacterium, as distinct from what it will tend to do.
Moving up the evolutionary ladder, does the introduction of nervous systems make any difference? What do we think about, say, nematodes or even horseshoe crabs? The feedback mechanisms underlying the self-regulatory processes in such animals may be leaps and bounds more sophisticated than in their non-neural forebears, but it's far from clear how such increasing complexity could introduce goals.
To cut to the chase, how can matters be any different with the members of Homo sapiens ? Looked at from a properly scientific point of view, is there any scope for the attribution of purposes or goals or the appraisal of our behaviour in any sense as right or wrong? I submit that a mere increase in complexity -even if by many orders of magnitude- does not turn the trick. To be clear, I'm not claiming there are no such facts -far from it- just that these facts cannot be articulated in the language of purely natural science.
The upshot
The foregoing thoughts are hardly original. David Hume is famous for having observed that ought cannot be derived from is:
(the issue, together with this quote are touched on with a different point of view in this post of lukeprog's). I think they need facing up to. I see three options:
Option (1): Accept the basic point, stick resolutely to naturalism, and deny that there are any facts as to what it is rational for any given member of Homo sapiens to do. In other words, become an eliminativist about all normative concepts. Paul Churchland, I understand, is an example of an exponent of this position (or something more nuanced along these lines).
Option (2): Reject the argument above on one or another ground. Try somehow to shoehorn normative facts into a naturalistic world-view, at the possible peril of the coherence of that world-view. I acknowledge that this can be a valiant undertaking for those whose commitments suggest it. Any who embark on it should be aware that there is at least a half a century's worth of beleaguered efforts to put this Humpty together -it is not an easy task. One might want to start with the likes of Ruth Garrett Millikan or Jerry Fodor, then their various critics.
Option (3): Accept the argument above and reconcile yourself to the existence of mutually incommensurable but indispensable understandings of yourself. Be happy.
One response
This is already ponderously long, but there is one response worth anticipating, namely, that oughts can be inferred from wants (or preferences or utility functions), and that wants (preferences, utility functions) are naturalistic. The problem here is with the second conjunct - wants (etc.) are not naturalistic. At least, not obviously so (and, incidentally, the same fate befalls beliefs). My explanation is as follows.
An example of the thinking behind this proposal presumably would be something like,
P) X's wanting that X eats an apple entails, other things being equal, that X ought to eat an apple.
The force of naturalism or physicalism in this context is presumably a commitment to some empirically testable analysis concerning wants comparable to "Water is H2O", e.g.
or
or ...
Now, if both of these thoughts (the thought about wants entailing oughts and the thought about there being an empirically testable analysis) are correct, then it should be possible to substitute the analysis into (P):
P') That X's brain is in state ABC entails, other things being equal, that X ought to eat an apple.
or
P'') That X is composed in part of some structure which implements a Turing machine which is in computational state DEF entails, other things being equal, that X ought to eat an apple.
or... (your favourite theory here)
I submit that neither P' nor P'' is at all plausible, for the reasons reviewed above. A thing's merely being in a certain physical or computational state does not imbue it with purpose. Such facts do not entail that there is anything which matters to it, or which makes anything right or wrong for the thing. Concerning P'', note that although computers are sometimes thought of as having purposes in virtue of our having designed them (the computer 'makes a mistake' when it calculates an incorrect value), there is not normally thought to be any sense in which they have intrinsic purposes independent of ours, as the view under scrutiny would require of them.
There are all kinds of possible refinements of P' and P'' (P' and P'' are commonly viewed as non-starters anyway, owing to the plausibility of so-called semantic externalism which they ignore). My question is whether any more refined option shows a way to defeat the objection being raised here.
Wants do indeed imply oughts. Since there plausibly is no physical or computational state being in which implies there is anything one ought to do, wanting is not identical to being in a physical or computational state.