I fully admit that I do not have strong outside-view evidence that this method will objectively improve your rationality-- if I did, I would post it. But many (most?) rationality techniques discussed here lack such evidence as well.

This seems like something that should be fixed. A few ideas:

• Scientific studies - seems to slow, requires dealing with academia... I think we can do better.
• Well designed self-experimentation - Gwern's studies on nootropics are the best examples I know of, but there are others like [Seth Robert's] self experiments.
• Studies done by a more formal organization - For example, I think CFAR might be doing studies like this.

Regarding my second point, it seems that this sort of thing could benefit a lot from a division of labor - where a small group of people design the experiments, and many more people just follow the instructions. It might be worth trying to organize a group of people willing to participate in these sorts of experiments, so that it is easier to test rationality techniques.