When words end in -ism, I tend to think of them as a framework or philosophy that is applicable in general terms, on a day-to-day basis. This has negative associations and examples (i.e. Darwinism, creationism), but on the whole I see it as something that applies strongly to what LWers mean by rationality, particularly instrumental rationality.
Mathematics might also be appropriately described that way, but not I think when it comes to mathematicians, who certainly use mathematics as a primary framework for their jobs, but may compartmentalize the rest of their daily activities and use non-mathematical approaches there. A lot of LW topics might appropriately be described as "mathematicism" in this sense, for example posts that give general day-to-day advice based on game theory or decision theory.
Edit: On further thought, my two examples actually don't match my stated criteria, since they're not about day-to-day activity. My revised explanation is that it has less to do with how often the ideas are used, and more to do with how much the person consciously identifies themselves with (or, in the case of "Darwinism", identifies other people with) the philosophy as a kind of group membership or point of pride.
I think rationality, and rationalists, fit under that -ism-ness as well.
I feel that the term "rationalism", as opposed to "rationality", or "study of rationality", has undesirable connotations. My concerns are presented well by Eric Drexler in the article For Darwin’s sake, reject "Darwin-ism" (and other pernicious terms):
So, my suggestion is to use "rationality" consistently and to avoid using "rationalism". Via similarity to "scientist" and "physicist", "rationalist" doesn't seem to have the same problem. Discuss.
(Typical usage on Less Wrong is this way already, 3720 Google results for "rationality" and 1210 for "rationalist", against 251 for "rationalism". I've made this post as a reference for when someone uses "rationalism".)