My impression was that Kaj's essay was not original to him but rather inspired by the paper he linked to at the bottom.

But in our case, we need God to say that the nerves in the skin are thermometers, the eyes, height measuring tools and so on.

Historically, it has been the other way round. We can recognise, without the hypothesis of God, that legs are good for walking, eyes for seeing, and so on, and these observable facts were taken as proof of the existence of a Designer.

Having dispensed with the Designer, we are left with the problem of explaining why living organisms appear to be made of distinct parts serving clear functions, and how we are able to say that these functions are sometimes performed well and sometimes badly, how we can describe some processes as pathological and some as healthy.

ETA: The answer isn't "evolution!", because when we use evolutionary techniques to solve computational problems, the result is typically something that works but we can't see how. When we look at living organisms we see things that work that we largely can see how. (The brain is a notable exception. Also, protein folding. But a heart is clearly a pump.)

Can you comment on how the concept of "ecological rationality" relates to this imaginary conversation?

Agreed. But it kind of means that some evolution of fallacies trending toward more complex argumentation patterns is taking place. Or? I'm not versed in the classics but I take it that they didn't have this large an (anti-)tool-set.

It is true that there are reasons for our biases; that human behavior was shaped by evolution and optimized for the natural environment. Many mistakes that we do are a result of behavior that contributes to survival in nature.

But I think that "contributes to survival" does not always lead to "solid inference rules". For example, imagine that a majority of the tribe is wrong about some factual question. (A question where being right or wrong is not immediately relevant for surviving.) It contributes to survival if an individual joins this majority, because it gets them allies. -- This could be excused by saying that in an ancient tribe without much specialization, a majority is more likely to be correct than an individual, therefore "follow the majority opinion" actualy is a good truth-finding heuristics. But that ignores the fact that people sometimes lie for a purpose, e.g. calumniate their opponents, or fabricate religious experience. So there is more in joining the majority than merely a decent truth-finding heuristics.

(EDIT: It's not like in the past humans lived in harmony with nature using their heuristics, and only today we have exploitable biases. People had exploitable biases even in the ancient environment -- their heuristics were correct often, but not always -- and people have exploited each other's biases even in the ancient environment. Not only we had adaptations to make mostly correct decisions, but also adaptations to exploit other people's flaws in the former adaptations.)

Also, no species is perfectly tuned to their environment. Some useful mutations simply didn't happen yet. Also, there are various trade-offs, so even if a species as a whole is optimized for given environment, some of their individual features may be suboptimal, as a price to improve other conflicting features. Therefore, assuming that every human bias is a result of a perfect behavior in the natural environment, would be assuming too much.

But otherwise, I like this.

A quote from that paper:

If a style of argumentation has survived critics for millennia, we can ask several questions: Could it be that there are evolutionary programs running in our heads that systematically push us to do the same things? Are those based on inferences that correlate with good fitness? Where does epistemic value differ from ecologic utility? Do the fallacists have some observation bias; do we suffer from the Focusing illusion (Schkade & Kahneman, 1998) when observing a bad argument?

I have this heard called the fallacy fallacy (though rational wiki sees that differently).

The one from Carnap ("Anything you can do, I can do meta") might not really be from Carnap. Can anyone find a source besides this one, which only gets it back to 1991?

I grew up in Long Island 20 miles from JFK airport. We could see the Concorde once in a while at JFK airport and if we were very lucky we would see it landing or taking off. The amount of mindspace in the world occupied by that beautiful plane was gigantic compared to that occupied by most other planes. Whether the Concorde was still a net deficit to the UK and France would require, I think, a calculation similar to figuring the deficit or surplus to the U.S. of putting people on the moon.

Kahneman suggests such an exercise for groups after pointing out that organizations generally act more rationally than individuals. The devil's advocate role and thinking at the worst possible outcome. We don't always have the luxury of having others near us for checking our thoughts. But we often have imaginary conversations with friends or parents. So it shouldn't be very difficult to assign a devil's advocate position to a imaginary voice. That should put in perspective the way we feel about the subject. It is a basic mean of delaying the strong coherence of the first good narrative.

Maybe it would be great to have an imaginary Bayesian friend...