There was a recent discussion on Facebook that led to an ask for a description of postrationality that isn't framed in terms of how it's different from rationality (or rather perhaps more a challenge that such a thing could not be provided). I'm extra busy right now until at least the end of the year so I don't have a lot of time for philosophy and AI safety work, but I'd like to respond with at least an outline of a constructive description of post/meta-rationality. I'm not sure everyone who identifies as part of the metarationality movement would agree with my construction, but this is what I see as the core of our stance.
Fundamentally I think the core belief of metarationality is that epistemic circularity (a.k.a. the problem of the criterion, the problem of perception, the problem of finding the universal prior) necessitates metaphysical speculation, viz. we can't reliably say anything about the world and must instead make one or more guesses to overcome at least establishing the criterion for assessing truth. Further, since the criterion for knowing what is true is unreliably known, we must be choosing that criterion on some other basis than truth, and so instead view that prior criterion as coming from usefulness to some purpose we have.
None of this is radical; it's in fact all fairly standard philosophy. What makes metarationality what it is comes from the deep integration of this insight into our worldview. Rather than truth or some other criteria, telos (usefulness, purpose) is the highest value we can serve, not by choice, but by the trap of living inside the world and trying to understand it from experience that is necessarily tainted by it. The rest of our worldview falls out of updating our maps to reflect this core belief.
To say a little on this, when you realize the primacy of telos in how you make judgments about the world, you see that you have no reason to privilege any particular assessment criterion except in so far as it is useful to serve a purpose. Thus, for example, rationality is important to the purpose of predicting and understanding the world often because we, through experience, come to know it to be correlated with making predictions that later happen, but other criteria, like compellingness-of-story and willingness-to-life, may be better drivers in terms of creating the world we would like to later find ourselves in. For what it's worth, I think this is the fundamental disagreement with rationality: we say you can't privilege truth and since you can't it sometimes works out better to focus on other criteria when making sense of the world.
So that's the constructive part; why do we tend to talk so much about postrationality by contrasting it with rationality? I think two reasons. One, postrationality is etiologically tied to rationality: the ideas come from people who first went deep on rationality and eventually saw what they felt were limitations of that worldview, thus we naturally tend to think in terms of how we came to the postrationalist worldview and want to show others how we got here from there. Second and relatedly, metarationality is a worldview that comes from a change in a person that many of us choose to identify with Kegan's model of psychological development, specifically the 4-to-5 transition, thus we think it's mainly worthwhile to explain our ideas to folks we'd say are in the 4/rationalist stage of development because they are the ones who can directly transition to 5/metarationality without needing to go through any other stages first.
Feel free to ask questions for clarification in the comments; I have limited energy available for addressing them but I will try my best to meet your inquiries. Also, sorry for no links; I wouldn't have written this if I had to add all the links, so you'll have to do your own googling or ask for clarification if you want to know more about something, but know that basically every weird turn of phrase above is an invitation to learn more.
Obviously. Which is why I said that the point was not any of the specific arguments in that debate - they were totally arbitrary and could just as well have been two statisticians debating the validity of different statistical approaches - but the fact that any two people can disagree about anything in the first place, as they have different models of how to interpret their observations.
This is very close to the distinction that I have been trying to point at; thank you for stating it more clearly than I managed to. The way that I'd phrase it is that there's a difference between considering a claim to be true, and considering its justification universally compelling.
It sounds like you have been interpreting me to say something like "Occam's Razor is false because its justification is not universally compelling". That is not what I have been trying to say. Rather, my claim has been "we can consider Occam's Razor true despite its justification not being universally compelling, but because there are no universally compelling justifications, we should keep trying out different justifications and seeing whether there are any that would seem to work even better".
If you say "but that's totally in line with 'Where Recursive Justification Hits Bottom' and the standard LW canon..." then yes, it is. That's my point. Especially since 'Recursive Justification' also says that we should just decide to believe in Occam's Razor, since it doesn't seem particularly useful to do otherwise, and because practically speaking, we don't have any better alternative:
As for my story about how the brain works: yes, it is obviously a vast simplification. That does not make it false, especially given that "the brain learns to use what has worked before and what it thinks is likely to make it win in the future" is exactly what Eliezer is advocating in the above post.
But what Eliezer also advocates in that post, is not elevating any rule - Occam's Razor included - into an unquestioned axiom, but to keep questioning even that, if you can:
I would say that there exists two kinds of metarationality: weak and strong. Weak metarationality is just standardly compatible with standard LW rationality, because of things like framing effects and self-fulfilling beliefs, as I have been arguing in other comments. But because the standard canon has given the impression that truth should be the only criteria for beliefs and missed the fact that there are plenty of beliefs that one can choose without violating Occam's Razor, this seems "metarational" and weird. Arguably like this shouldn't be called meta/postrationality in the first place, because it's just standard rationality.
The way to phrase strong metarationality might be, is that classic LW rationality is what you get when you take a specific set of axioms as your starting point, and build on top of that. Metarationality is what you get when you acknowledge that this does indeed seem like the right thing to do most of the time, but that we should also be willing (as Eliezer advocates above) to question that, and try out different starting axioms as well, to see whether there would be any that would be even better.
In my experience, strong metarationality isn't useful because it would point to any basic axioms that would be better than LW's standard ones - if it does, I haven't found any, and the standard assumptions continue to be the most useful ones. But what does make it somewhat useful is in that when you practice questioning everything, and e.g. distinguishing between "Occam's Razor is true" and "I have assumed Occam's Razor to be true because that seems useful", then that helps in catching assumptions which don't fall directly out of the standard axioms, and which you've just assumed to be true without good justification.
E.g. "my preferred system of government is the best one" is a belief that should logically be assigned much lower confidence than "Occam's Razor is true"; but the brain only has limited precision in assigning credence values to claims. So most people have beliefs which are more like the government one than the Occam's Razor one, despite being assigned a similar level of credence as the Occam's Razor one is. By questioning and testing even beliefs which are like Occam's Razor, one can end up questioning and revising beliefs which actually should be questioned, which one might never have questioned otherwise. This is valuable even if the Occam's Razor-like beliefs survive that questioning unscathed - but the exercise does not work unless one actually does make a serious attempt to question them.