Okay, point taken. Individual behavior would become irrational, even if the thought processes behind the actions were perfectly rational. I will note that our beliefs tend to follow our actions. Trying to act irrationally likely has corrosive effects on our capacity to be rational unless we're very careful.
Now the question is: are there any benefits to this sort of group rationality, as opposed to simply encouraging everyone in the group to become more individually rational?
I don't know, but I think the legal system might be (some) evidence in favor of using roles.
If cases were judged by professionals encouraged to be individually rational, it's not clear how the cases would turn out differently, but there don't seem to be as many corrective mechanisms against small biases. It would be something like open-loop control.
In Reply to: Rationalization, Epistemic Handwashing, Selective Processes
Eliezer Yudkowsky wrote about scientists defending pet hypotheses, and prosecutors and defenders as examples of clever rationalization. His primary focus was advice to the well-intentioned individual rationalist, which is excellent as far as it goes. But Anna Salamon and Steve Rayhawk ask how a social system should be structured for group rationality.
The adversarial system is widely used in criminal justice. In the legal world, roles such as Prosecution, Defense, and Judge are all guaranteed to be filled, with roughly the same amount of human effort applied to each side. Suppose individuals chose their own roles. It is possible that one role turns out more popular. Because different effort is applied to different sides, selecting for the positions with the strongest arguments will no longer much select for positions that are true.
One role might be more popular because of an information cascade: individuals read the extant arguments and then choose a role, striving to align themselves with the truth, and create arguments for that position. Alternately, a role may be popular due to status-based affiliation, or striving to be on the "winning" side.
I'm well aware that there are vastly more than two sides to most questions. Imagine a list of rationalist roles something like IDEO's "Ten Faces".
Example rationalist roles, leaving the obvious ones for last:
Due to natural group phenomena (cascades, affiliation), in order to achieve group rationality, there need to be social structures that strive to prevent those natural phenomena. Roles might help.