To the extent that irrationality is a result of compartmentalization, this may be the same thing as creating a way to measure how effectively you are accomplishing your goals, which is going to vary between people depending on what their goals are.

For most interesting goals I can think of, creating a rigorous quantitative measure is next to impossible. However, there are a few goals, like running a mile in under four minutes, that lend themselves well to this approach. Perhaps SI could find a group of individuals engaged in such a goal and offer their services as rationality consultants?