This seems reasonable, but I have to ask about "correctly describes". The statement

"want to eat an apple" implies being in brain state ABC or computational state DEF (or something of that nature)

is plausible to me. I think the reverse implication, though raises a problem:

being in brain state ABC or computational state DEF (or something of that nature) implies "want to eat an apple"

But maybe neither of these is what you have in mind?

You use a fair bit of normative, teleological vocabulary, here: 'purpose', 'goal', 'success', 'optimisation', 'trying', being 'good' at 'steering' the future. I understand your point is that these terms can all be cashed out in unproblematic, teleonomic terms, and that this is more or less an end of the matter. Nothing dubious going on here. Is it fair to say, though, that this does not really engage my point, which is that such teleonomic substitutes are insufficient to make sense of rationality?

To make sense of rationality, we need claims such as,

• One ought to rank probabilities of events in accordance with the dictates of probability theory (or some more elegant statement to the effect).

If you translate this statement, substituting for 'ought' the details of the teleonomic 'ersatz' correlate, you get a very complicated statement about what one likely will do in different circumstances, and possibly about one's ancestor's behaviours and their relation to those ancestors' survival chances (all with no norms).

This latter complicated statement will not mean what the first statement means, and won't do the job required in discussing rationality of the first statement. The latter statement will be an elaborate description; what's needed is a prescription.

Probably none of this should matter to someone doing biology, or for that matter decision theory. But if you want to go beyond and commit to a doctrine like naturalism or physical reductionism, then I submit this does become relevant.

I would taboo and translate that use to yield something like "To make sense of rationality in an agent, one needs to accept/assume/stipulate that the agent sometimes acts with a purpose in mind. We need to understand 'purpose', in that sense, to understand rationality."

Thanks, yes. This is very clear. I can buy this.

But I think I understand this kind of purpose, identifying it as the cognitive version of something like "being instrumental to survival and reproduction". That is, it is possible for an outside observer to point to behaviors or features of a virus that are instrumental to viral survival and reproduction.

Sorry if I'm slow to be getting it, but my understanding of your view is that the sort of purpose that a bacterium has, on the one hand, and the purpose required to be a candidate for rationality, on the other, are, so to speak, different in degree but not in kind. They're the same thing, just orders of magnitude more sophisticated in the latter case (involving cognitive systems). This is the idea I want to oppose. I have tried to suggest that bacterial purposes are 'merely' teleonomic -to borrow the useful term suggested by timtyler- but that human purposes must be of a different order.

Here's one more crack at trying to motivate this, using very evidently non-scientific terms. On the one hand, I submit that you cannot make sense of a thing (human, animal, AI, whatever) as rational unless there is something that it cares about. Unless that is, there is something which matters or is important to it (this something can be as simple as survival or reproduction). You may not like to see a respectable concept like rationality consorting with such waffly notions, but there you have it. Please object to this if you think it's false.

On the other hand, nothing in nature implies that anything matters (etc) to a thing. You can show me all of the behavioural/cognitive correlates of X's mattering to a thing, or of a thing's caring about X, and provide me detailed evolutionary explanations of the behavioural correlates' presence, but these correlates simply do not add up to the thing's actually caring about X. X's being important to a thing, X's mattering, is more than a question of mere behaviour or computation. Again, if this seems false, please say.

If both hands seem false, I'd be interested to hear that, too.

At the level of a bacterium, there are second-messenger chemicals that symbolize or represent situations that are instrumental to survival and reproduction. At the level of the nematode, there are neuron firings serving as symbols. At the level of a human the symbols can be vocalizations: "I'm horny; how about you?". I don't see anything transcendently new at any stage in this progression, nor in the developmental progression that I offered as a substitute.

As soon as we start to talk about symbols and representation, I'm concerned that a whole new set of very thorny issues get introduced. I will shy away from these.

Let me try putting that in different words: "Norms are in the eye of the beholder. Natural science tries to be objective - to avoid observer effects. But that is not possible when studying rationality. It requires a different, non-reductionist and observer dependent way of looking at the subject matter." If that is what you are saying, I may come close to agreeing with you. But somehow, I don't think that is what you are saying.

"It requires a different, non-reductionist ... way of looking at the subject matter." -I can agree with you completely on this. (I do want however to resist the subjective, "observer dependent" part )

Thanks for this - I hadn't encountered this concept. Looks very useful.

I'm also saying that non-natural "facts" are as easy to work with as natural facts. The issue that they're not part of natural science doesn't impact our ability to discuss them productively.

I agree entirely with this. This exercise isn't meant in any way to be an attack on decision theory or the likes. The target is so-called naturalism - the view that all facts are natural facts.

I appreciate this clarification. The point is meant to be about, as you say, the empirical sciences. I agree that there can be arbitrarily sophisticated scientific theories concerning rational behaviour - just that these theories aren't straight-forwardly continuous with the theories of natural science.

In case you haven't encountered it and might be interested, the underdetermination problem associated with inferring to beliefs and desires from mere behaviour has been considered in some depth by Donald Davidson, eg in his Essays on Actions and Events

I appreciate your efforts to spell things out. I have to say I'm getting confused, though

You started by making an argument that listed a series of stages (virus, bacterium, nematode, man) and claimed that at no stage along the way (before the last) were any kind of normative concepts applicable.

I meant to say that at no stage -including the last!- does the addition of merely naturalistic properties turn a thing into something subject to norms -something of which it is right to say it ought, for its own sake, to do this or that.

I also said that the sense of right and wrong and of purpose which biology provides is merely metaphorical. When you talk about "the illusion of teleology in nature", that's exactly what I was getting at (or so it seems to me). That is, teleology in nature is merely illusory, but the kind of teleology needed to make sense of rationality is not - it's real. Can you live with this? I think a lot of people are apt to think that illusory teleology sort of fades into the real thing with increasing physical complexity. I see the pull of this idea, but I think it's mistaken, and I hope I've at least suggested that adherents of the view have some burden to try to defend it.

Do you believe it is possible to tell a teenager what she "ought" to do?

Now that is a whole other can of worms...

At what stage in development do normative judgements become applicable.

This is a fair and a difficult question. Roughly, another individual becomes suitable for normative appraisal when and to the extent that s/he becomes a recognizably rational agent -ie, capable of thinking and acting for her/himself and contributing to society (again, very roughly). All kinds of interesting moral issues lurk here, but I don't think we have to jump to any conclusions about them.

In case I'm giving the wrong impression, I don't mean to be implying that people are bound by norms in virtue of possessing some special aura or other spookiness. I'm not giving a theory of the nature of norms - that's just too hard. All I'm saying for the moment is that if you stick to purely natural science, you won't find a place for them.

Rational-ought beliefs and actions are the ones optimal for achieving your goals. Goals and optimallity can be explained in scientific language. Rational-ought is not moral-ought. Moral-ought is harder to explain because it about the goals an agent should have,not the ones they happen to have.

I'm sincerely not sure about the rational-ought/moral-ought distinction - I haven't thought enough about it. But anyway, I think moral-ought is a red herring, here. As far as I can see, the claims made in the post apply to rational-oughts. That was certainly the intention. In other posts on LW about fallacies and the details of rational thinking, it's a commonplace to use quite normative language in connection with rationality. Indeed, a primary goal is to help people to think and behave more rationally, because this is seen for each of us to be a good. 'One ought not to procrastinate', 'One ought to compensate for one's biases', etc..

Try somehow to shoehorn normative facts into a naturalistic world-view, at the possible peril of the coherence of that world-view.

Easily done with non-moral norms such as rationality.

Would love to see the details... :-)

Not if you think of purpose as a metaphysical fundamental. Easily, if a purpose is just a particular idea in the mind. If I intend to but a lawnmower, and I write "buy lawnmower" on a piece of paper, there is nothing mysterious about the note, or about the state of mind that preceded it.

I'm not sure I get this. The intention behind drawing the initial distinction between is/ought problems was to make clear the focus is not on, as it were, the mind of the beholder. The question is a less specific variant of the question as to how any mere physical being comes to have intentions (e.g., to buy a lawnmower) in the first place.

That you want to do something does not mean you ought to do it in the categorial, unconditional sense of moral-ought.

I agree, but I think it does mean you ought to in a qualified sense. Your merely being in a physical or computational state, however, by itself doesn't, or so the thought goes.

If I bet higher than 1/6th on a fair die's rolling 6 because in the last ten rolls 6 hasn't come up -meaning it's now 'due'- I make a mistake. I commit an error of reasoning; I do something wrong; I act in a manner I ought not to.

What about the virus particle which, in the course of sloshing about in an appropriate medium, participates in the coming into existence of a particle composed of RNA which, as it happens, is mostly identical but differs from itself in a few places. Are you saying that this particle makes a mistake in the same sense of 'mistake' as I do in making my bet?

Option (1): The sense is precisely the same (and it is unproblematically naturalistic). In this case I have to ask what the principles are by which one infers to conclusions about a virus's mistakes from facts about replication. What are the physical laws, how are their consequences (the consequences, again, being claims about what a virus ought to do) measured or verified, and so on?

Option (2): The senses are different. This was the point of calling the RNA mistake metaphorical. It was to convey that the sense is importantly different than it is in the betting case. The idea is that the sense, if any, in which a virus makes a 'mistake' in giving rise to a non-exact replica of itself is not enough to sustain the kind of norms required for rationality. It is not enough to sustain the conclusions about my betting behaviour. Is this fair?

Suppose I hear Bob say "I want to eat an apple." Am I justified in assigning a higher probability to "Bob wants to eat an apple" after I hear this than before (assuming I don't have some other evidence to the contrary, like someone is holding a gun to Bob's head)?

I think this question hits the nail on the head. You are justified in assigning a higher probability to "Bob wants to eat an apple" just in case you are already justified in taking Bob to be a rational agent (other things being equal...). If Bob isn't at least minimally rational, you can't even get so far as construing his words as English, let alone to trust that his intent in uttering them is to convey that he wants to eat an apple (think about assessing a wannabe AI chatbot, here). But in taking Bob to be rational, you are already taking him to have preferences and beliefs, and for there to be things which he ought or ought not to do. In other words, you have already crossed beyond what mere natural science provides for. This, anyway, is what I'm trying to argue.