Here is a segment of the line of reasoning that I have been considering lately.

• The details of the creation of fAI are unknown; they must be discovered if we wish to avert the forecast catastrophe and reap the full benefits of AI.

• Further insights into decision-making are necessary for the creation of fAI.

• Teaching can be used as an efficient means of learning.

• Teaching rationality is expected to produce insights into decision-making.

• Insights into decision-making are what is missing from an understanding of fAI, AFAIK.

Furthermore, there will be significant amount of disutility before fAI is implemented. As utility is that thing which we wish to maximize, we ought to want to want to decrease disutility. The primary means we have at our disposal is rationality.

The caveat is that we must make sure that the benefits of teaching rationality in the present exceed the cost from delaying fAI.