Yudkowsky says:
So if you understand what concept we are generally getting at with this word "rationality", and with the sub-terms "epistemic rationality" and "instrumental rationality", we have communicated: we have accomplished everything there is to accomplish by talking about how to define "rationality". What's left to discuss is not what meaning to attach to the syllables "ra-tio-na-li-ty"; what's left to discuss is what is a good way to think.
With that said, you should be aware that many of us will regard as controversial - at the very least - any construal of "rationality" that makes it non-normative:
For example, if you say, "The rational belief is X, but the true belief is Y" then you are probably using the word "rational" in a way that means something other than what most of us have in mind. (E.g. some of us expect "rationality" to be consistent under reflection - "rationally" looking at the evidence, and "rationally" considering how your mind processes the evidence, shouldn't lead to two different conclusions.) Similarly, if you find yourself saying "The rational thing to do is X, but the right thing to do is Y" then you are almost certainly using one of the words "rational" or "right" in a way that a huge chunk of readers won't agree with.
A normative belief in rationality is, as far as I can tell, not possible for someone who does not have a clear concpetion of what rationality is. I am trying to present tools for forming such a conception. The theory I am presenting is, most accurately, a rationally constructed language, not a prescriptive theory on whether it is moral to be rational. The merit of this language is that it should allow you to converse about rationality with mysticists or religious people so that you both understand what you are talking about. It seems to me the ID vs. evolution debate remains unresolved among the general public (in the USA) because neither side has managed to speak the same language as the other side. My language is not formally defined in the sense of being a formal language, but it has formally defined ontological types.
ID vs. evolution debate remains unresolved among the general public (in the USA) because neither side has managed to speak the same language as the other side
Those are the labels used to describe the issue by the participants. But taking an outside view, the issue is inconsistent principles between the two sides. The fact that true religious believers reject the need for beliefs to pay rent in anticipated experience won't be solved by new vocabulary.
I've seen there's discussion on LW about rationality, namely, about what it means. I don't think a satisfactory answer can be found without defining what rationality is not. And this seems to be a problem. As far as I know, rationality on LW does not include systematic methods for categorizing and analyzing irrational things. Instead, the discussion seems to draw a circle around rationality. Everyone on LW is excepted to be inside this circle - think of it as a set in a Venn diagram. On the border of the circle there is a sign saying: "Here be dragons". And beyond the circle there is irrationality.
How can we differentiate the irrational from the rational, if we do not know what the irrational is?
But how can we approach the irrational, if we want to be rational?
It seems to me there is no way to give a satisfactory account of rationality from within rationality itself. If we presuppose rationality is the only way to attain justification, and then try to find justification for rationalism (the doctrine according to which we should strive for rationality), we are simply making a circular argument. We already presupposed rationalism before trying to find justification for doing so.
Therefore it seems to me we ought to make a metatheory of rationality in order to find out what is rational and what is irrational. The metatheory itself has to be as rational as possible. That would include having an analytically defined structure, which permits us to at least examine whether the metatheory is logically consistent or inconsistent. This would also allow us to also examine whether the metatheory is mathematically elegant, or whether the same thing could be expressed in a simpler form. The metatheory should also correspond with our actual observations so that we could figure out whether it contradicts empirical findings or not.
How much interest is there for such a metatheory?